Method and system for identifying missing test scenarios by comparing authorized processes with available test scenarios

ABSTRACT

Identifying missing test scenarios based on authorization policies, including: analyzing the authorization policies applied to non-super-users of a specific data-oriented large-scale software system in order to identify organization-specific in-use scenarios, whereby the non-super-users are usually unauthorized to irrelevant operations; comparing the identified organization-specific in-use scenarios with available test scenarios repairable by a non-programmer user, each available test scenario repairable by a non-programmer user tests at least two transactions; identifying organization-specific in-use scenarios that do not have corresponding test scenarios repairable by a non-programmer user, referred to as first set of missing test scenarios; and providing at least one of the missing test scenarios to a non-programmer user, having proper authorizations, for generating a test scenario repairable by a non-programmer user that covers the at least one missing test scenario.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 61/375,858, filed Aug. 22, 2010.

The following US Patent Applications, filed on May 8, 2011, may includerelated subject matter: Ser. Nos. 13/103,075; 13/103,076; 13/103,078;13/103,080; 13/103,081; 13/103,082; 13/103,083 (now U.S. Pat. No.8,522,083); Ser. No. 13/103,086; and Ser. No. 13/103,088.

TECHNICAL FIELD

Some of the disclosed embodiments relate to testing of computersoftware, and more specifically to method and system for identifyingmissing test scenarios by comparing authorized processes with availabletest scenarios.

BACKGROUND

Software testing is a fundamental component of software qualityassurance and includes many methodologies and types, such as acceptancetesting, black box testing, compatibility testing, functional testing,integration testing, regression testing, system testing, unit testing,and white box testing. It is common for software developers to spend asignificant amount of time and effort on generating the tests, and tospend a significant amount of time and effort on maintaining theavailable tests.

Common testing methodologies, such as manual testing and record and playtesting, require the test setup to be almost the same as the originalsetup, which is an unrealistic requirement in many cases. This isespecially noticeable when facing a system upgrade, where all thestandard components are upgraded without additional developments ormodules. The effect of the system upgrade on the user code is unknownand thus should be tested.

BRIEF SUMMARY

In one embodiment, a computer implemented method for identifying missingtest scenarios based on authorization policies, comprising: analyzingthe authorization policies applied to non-super-users of a specificdata-oriented large-scale software system in order to identifyorganization-specific in-use scenarios; whereby the non-super-users areusually unauthorized to irrelevant operations; comparing the identifiedorganization-specific in-use scenarios with available test scenariosrepairable by a non-programmer user, each available test scenariorepairable by a non-programmer user tests at least two transactions;identifying organization-specific in-use scenarios that do not havecorresponding test scenarios repairable by a non-programmer user,referred to as first set of missing test scenarios; and providing atleast one of the missing test scenarios to a non-programmer user, havingproper authorizations, for generating a test scenario repairable by anon-programmer user that covers the at least one missing test scenario.Optionally, the data-oriented large-scale software system is Oracle ERPand the authorization policies are taken from Oracle Applications UserManagement and Oracle Applications Responsibility Management.Optionally, the embodiment further comprising identifying the formsexposed to the non-super-users by analyzing the responsibilities fromOracle Applications Responsibility Management. Optionally, thetransactions are selected from the group of forms, workflow, reports,concurrent programs, and other executable units. Optionally, thegenerated test scenario repairable by a non-programmer is a parametrictest scenario repairable by a non-programmer user, which definesdifferent alternatives for executing the test scenario. Optionally, theparametric test scenario repairable by a non-programmer user furtherdefines different alternatives for evaluating execution outcome, and isexpressed in terms that substantially conform to terms used in thenon-programmer user's work environment, and coherent with actual manualoperation of the test scenario. Optionally, the embodiment furthercomprising identifying additional organization-specific in-use scenariosby registering the executed transactions and registering order ofexecuting the transactions. Optionally, the embodiment furthercomprising identifying the additional organization-specific in-usescenarios by registering context of executing the transactions.Optionally, generating the test scenario repairable by a non-programmeruser comprises recording the non-programmer user executing the missingtest scenario.

In one embodiment, a computer program product for use in a computer toidentify missing test scenarios based on authorization policies, thecomputer including a processor, and the computer program productcomprising: program code for analyzing the authorization policiesapplied to non-super-users of a specific data-oriented large-scalesoftware system in order to identify organization-specific in-usescenarios; whereby the non-super-users are usually unauthorized toirrelevant operations; program code for comparing the identifiedorganization-specific in-use scenarios with available test scenariosrepairable by a non-programmer user, each available test scenariorepairable by a non-programmer user tests at least two transactions;program code for identifying organization-specific in-use scenarios thatdo not have corresponding test scenarios repairable by a non-programmeruser, referred to as first set of missing test scenarios; and programcode for providing at least one of the missing test scenarios to anon-programmer user, having proper authorizations, for generating a testscenario repairable by a non-programmer user that covers the at leastone missing test scenario. Optionally, the data-oriented large-scalesoftware system is Oracle ERP and the authorization policies are takenfrom Oracle Applications User Management and Oracle ApplicationsResponsibility Management. Optionally, the embodiment further comprisingprogram code for identifying the forms exposed to the non-super-users byanalyzing the responsibilities from Oracle Applications ResponsibilityManagement. Optionally, the transactions are selected from the group offorms, workflow, reports, concurrent programs, and other executableunits. Optionally, the generated test scenario repairable by anon-programmer is a parametric test scenario repairable by anon-programmer user, which defines different alternatives for executingthe test scenario. Optionally, the parametric test scenario repairableby a non-programmer user further defines different alternatives forevaluating execution outcome, and is expressed in terms thatsubstantially conform to terms used in the non-programmer user's workenvironment, and coherent with actual manual operation of the testscenario. Optionally, the embodiment further comprising program code foridentifying additional organization-specific in-use scenarios byregistering the executed transactions and registering order of executingthe transactions. Optionally, the embodiment further comprising programcode for identifying the additional organization-specific in-usescenarios by registering context of executing the transactions.Optionally, generating the test scenario repairable by a non-programmeruser comprises recording the non-programmer user executing the missingtest scenario.

In one embodiment, a computer system to identify missing test scenariosbased on authorization policies, the computer system comprising: astorage device containing computer usable code; and a processing unitoperable to execute the computer usable code to: analyze theauthorization policies applied to non-super-users of a specificdata-oriented large-scale software system in order to identifyorganization-specific in-use scenarios; whereby the non-super-users areusually unauthorized to irrelevant operations; compare the identifiedorganization-specific in-use scenarios with available test scenariosrepairable by a non-programmer user, each available test scenariorepairable by a non-programmer user tests at least two transactions;identify organization-specific in-use scenarios that do not havecorresponding test scenarios repairable by a non-programmer user,referred to as first set of missing test scenarios; and provide at leastone of the missing test scenarios to a non-programmer user, havingproper authorizations, to generate a test scenario repairable by anon-programmer user that covers the at least one missing test scenario.Optionally, the data-oriented large-scale software system is Oracle ERPand the authorization policies are taken from Oracle Applications UserManagement and Oracle Applications Responsibility Management.Optionally, the processing unit further executes the computer usablecode to identify the forms exposed to the non-super-users by analyzingthe responsibilities from Oracle Applications Responsibility Management.Optionally, the transactions are selected from the group of forms,workflow, reports, concurrent programs, and other executable units.Optionally, the generated test scenario repairable by a non-programmeris a parametric test scenario repairable by a non-programmer user, whichdefines different alternatives for executing the test scenario.Optionally, the parametric test scenario repairable by a non-programmeruser further defines different alternatives for evaluating executionoutcome, and is expressed in terms that substantially conform to termsused in the non-programmer user's work environment, and coherent withactual manual operation of the test scenario. Optionally, the processingunit further executes the computer usable code to register the executedtransactions and register order of executing the transactions in orderto identify additional organization-specific in-use scenarios.Optionally, the processing unit further executes the computer usablecode to register context of executing the transactions.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are herein described, by way of example only, withreference to the accompanying drawings. In the drawings:

FIG. 1A, FIG. 1B, and FIG. 1C illustrate embodiments of systems fortesting a data-oriented large-scale software package;

FIG. 1D illustrates one embodiment of sorting impact analysis resultsaccording to the usability of the impacted transactions in a specificorganization;

FIG. 1E illustrate one embodiment of system components useful forimplementing various embodiments;

FIG. 2A and FIG. 2B illustrate embodiments for identifyingnon-executable human-readable test scenarios to be updated due to codechanges;

FIG. 2C illustrates one embodiment of a computer system to identifynon-executable human-readable test scenarios to be updated due to codechanges;

FIG. 3A illustrates one embodiment of a method for identifying missingtest scenarios based on a repository of organization-specific in-usescenarios;

FIG. 3B illustrates one embodiment of a computer system to identifymissing test scenarios based on a repository of organization-specificin-use scenarios;

FIG. 3C illustrates one embodiment of a method for identifying missingtest scenarios based on authorization policies;

FIG. 3D illustrates one embodiment of a computer system to identifymissing test scenarios based on authorization policies;

FIG. 4 illustrates one embodiment of a record and play process;

FIG. 5A, FIG. 5B, and FIG. 5C illustrate three instances of the samescenario, which result in different screen flows;

FIG. 5D illustrates a master scenario including the various identifiedscreens;

FIG. 6A illustrates one embodiment of a method for determining whetherto correct a failed test scenario using re-recording or code change;

FIG. 6B illustrates one embodiment of a method for processing failuresof executing test scenarios over a data-oriented large-scale softwaresystem;

FIG. 6C illustrates one embodiment of a method for correcting testscenarios using re-recording;

FIG. 6D illustrates one embodiment of a method for automaticauthorization-based routing of failed test scenarios;

FIG. 7 illustrates one embodiment of a method for semiautomaticexecution of a functioning test scenario to enable a non-programmer userto generate a new test scenario based on the functioning test scenario;

FIG. 8A, FIG. 8B, and FIG. 8C illustrate a method for semiautomaticexecution of a functioning test scenario to enable a non-programmer userto generate a new test scenario based on the functioning test scenario;

FIG. 9 illustrates one embodiment of a method for improving userfriendliness of a non-executable human-readable test scenario byremoving unessential test steps;

FIG. 10A and FIG. 10B illustrate one embodiment of a method forimproving the user friendliness of a non-executable human-readable testscenario by removing unessential test steps;

FIG. 11 illustrates one embodiment of a method for user-friendly testscenario recording;

FIG. 12 illustrates one embodiment of a method for directing the usersas to what software modules need to be checked when facing a codechange;

FIG. 13 is a schematic illustration of a failure analysis component;

FIG. 14 is a schematic illustration of the likelihood of failure andrisk assessment components;

FIG. 15A illustrates one embodiment of displaying test scenariosaccording to their importance;

FIG. 15B illustrates one embodiment of displaying test scenariosaccording to importance versus the number of users;

FIG. 15C illustrates one embodiment of displaying test scenariosaccording to importance versus type of the test scenarios; and

FIG. 15D illustrates one embodiment of displaying test scenariosaccording to related modules versus estimated upgrade complexity.

DETAILED DESCRIPTION

The term “transaction” is defined as a computer program, such as SAP ERPtransaction, or Oracle Application Form.

The term “business process”, also known as “workflow”, is defined as aset of coordinated tasks and activities, conducted by both people andequipment, which leads to accomplish a specific organizational goal. Abusiness process usually executes two or more transactions; however, insome embodiments a business process may execute only one transaction.

The term “test scenario” is defined as a user-interaction description offunctional and/or technical specification for testing a business processand/or one or more transactions. At least 50% of the test scenarios areexpressed by user interaction activities and not code-related language,such as procedure calls. A test scenario may specify generally how totest one or more business processes, business requirements, testconditions, logical data, and/or expected results. Additionally oralternatively, a test scenario may detail the exact steps to be followedto complete the testing, such as a sequence of test steps with roles andapplication used, data used for testing, expected results, andcontextual information. Usually, many different test scenarios areneeded to test a single business process completely. A scenario executesone or more transactions, and many different scenarios may relate to asingle business process.

In some embodiments, the test scenarios are non-executablehuman-readable test scenarios. It is noted that the non-executablehuman-readable test scenarios are significantly different from unittests and test cases, which are executable test descriptions written byprogrammers.

The term “parametric test scenario repairable by a non-programmer user”is defined as a test scenario pseudocode written with expressions thatsubstantially conform to the user's work environment and therefore isunderstandable by a non-programmer user.

The term “organization-specific in-use scenarios” or shortly, “in-usescenarios”, in relation to a specific organization, is defined as asubset of the scenarios that are actually executed by the specificorganization, and therefore are relevant to the specific organization.The term “organization-specific in-use scenarios” is specificallylimited to include less than half the total possible scenarios supportedby the data-oriented large-scale software system operated by thespecific organization. A subset of scenarios that includes more thanhalf the total scenarios supported by the data-oriented large-scalesoftware system of the specific organization is not considered herein as“organization-specific”. In some cases, organization-specific in-usescenarios are specifically limited to include less than 10% the totalpossible scenarios supported by the data-oriented large-scale softwaresystem of the specific organization. In some examples, theorganization-specific in-use scenarios may be substantially limited to asubset of the actually executed scenarios (i) used by certain users orcertain departments of the specific organization, (ii) potentiallyerror-prone scenarios or transactions identified by a failure loggermonitoring the specific organization, and/or (iii) manually identifiedas important to the specific organization, or as important to similarorganizations in the same field.

The term “organization-specific in-use software elements” refers tosoftware elements executed by the organization-specific in-usescenarios.

The terms “different installations” or “other installations” or “old andnew installations”, in relation to certain test scenario(s), are definedas software releases and/or software configurations that cannot exchangeplain scenario recordings and still successfully play the exchangedrecordings of the certain test scenario without some adaptation.

For example, a first company and a second company are considered tooperate different installations of data-oriented large-scale softwaresystems, in relation to a certain test scenario, if a recording of thecertain scenario taken on the installation of the first company cannotplay successfully on the installation of the second company without someadaptation of the recorded scenario to the specific installation of thesecond company. Examples of different installations include cases wherethe companies run (i) exactly the same ERP software releases, with thesame customizations, but with different data, (ii) exactly the same ERPsoftware releases, with different customizations, and/or (iii) differentERP software releases.

Examples of data-oriented large-scale software systems includeEnterprise Resource Planning (ERP), such as from the followingtrademarks: SAP, Oracle Application, The Sage Group, Microsoft Dynamics,and SSA Global Technologies; billing systems (also known as revenuemanagement) such as from the following trademarks: Amdocs, ComverseInc., and Convergys Corporation; service management systems; andportfolio management systems.

The term “user-generated code” is defined as code written for adata-oriented large-scale software system, which changes the behavior ofthe data-oriented large-scale software system, and was not generated andusually unsupported by the manufacturer of the data-oriented large-scalesystem. User-generated code is usually created for a specificorganization. User-generated code can be implemented as an independenttransaction or as an enhancement to an existing transaction. Examples ofuser-generated code in ERP include, but are not limited to, adding codeto a “user-exit” in SAP environment, changing the original ERP systemcode, changing the visible/invisible property of a field inside awindow, and defining a field as mandatory. It is noted that activitiesthat do not change the system behavior, such as changing userpermissions, or data changes such as setting a new catalog number oradding a warehouse, are not considered as user-generated code.

The term “non-programmer user” is defined as a user familiar with thedata-oriented large-scale software system. The non-programmer user knowshow to operate the system, may be able to draft requirements for newbusiness processes, but does not prepare the actual user-generated codeused to customize the data-oriented large-scale software system.Examples of non-programmer users include ERP business analyst, ERPprofessional, business expert, and/or end-user.

The term “customization” in the context of a data-oriented large-scalesoftware system refers to data describing the business processes, suchas SAP Configuration Tables and Oracle Form Personalization. The term“customization change” refers to change to data structures related tothe business processes such as changes to SAP Configuration Tables orOracle Form Personalization.

The term “abnormal termination” includes errors such as an internalsystem error, an activation error, and/or a compilation error. Abnormaltermination does not include normal termination with a value differentfrom an expected value.

FIG. 1A illustrates one embodiment of a system for testing adata-oriented large-scale software package, such as ERP or billingsoftware platforms. A non-programmer user 100 records test scenariosthat are converted to parametric test scenarios 104 by a test recordingand analysis component 102. The parametric test scenarios 104 describehow to execute the test scenarios and how to evaluate the executionoutcomes. Optionally, the execution outcome is evaluated by comparingthe results of the executed test scenarios with the results of therecorded scenarios.

In-use identifier component 114.

An in-use identifier component 114 identifies scenarios that areactually in use by the specific organization, referred to as in-usescenarios 116. In some embodiments, the in-use identifier component 114receives inputs from one or more of the following: a usage logger 110that registers the used scenarios and/or transactions, a bug logger 111that registers faults in the used scenarios and/or transactions, and/orfrom a non-programmer user 100 that identifies the used scenarios and/ortransactions. Optionally, the in-use identifier component 114 assignsweights to the various input scenarios according to their importance tothe organization, as further discussed below.

In one embodiment, the in-use identifier component 114 identifiestransactions, optionally because the usage logger 110 and the bug logger111 register transactions and the non-programmer user 100 identifiestransactions and not scenarios. As a result, the in-use scenarios 116may include all the scenarios operated by the identified transactions,and therefore may include redundant scenarios. In one embodiment, thein-use identifier component 114 identifies the used procedures, and thenthe used procedures are mapped to the in-use scenarios 116.

Code change analysis component 124 and code-to-scenario component 125.

The code change analysis component 124 receives as inputs an old codeversion 122 and a new code version 123, and identifies code that waschanged or impacted as a result of replacing the old code version 122with the new code version 123. In one example, the code change analysiscomponent 124 is based on static impact analysis, and/or text comparisonalgorithm. In the field of code change analysis, the old code version122 may also be known as the base code or standard code, which mayinclude user-generated code. The new code version 123 may result, forexample, from a software upgrade process, from a direct code change by aprogrammer (that may be related or unrelated to an upgrade process),and/or from adding/removing/replacing a certain code module. Theidentified changed code is received by a code-to-scenarios component125. Code-to-scenarios component 125 identifies the list of changedscenarios 126, which includes business processes that use at least someof the changed code or some of the code affected by the code change.

In some embodiments, the code change analysis component 124 may also beused to identify customization changes by comparing an old customizationversion with a new customization version. In these embodiments, thecode-to-scenarios component 125 operates as a customization-to-scenariocomponent, which identifies the list of changed scenarios 126, whichincludes business processes that use at least some of the changedcustomizations.

In one embodiment, the code-to-scenarios component 125 identifies thesoftware elements operated by each business process using a staticanalysis, such as control flow analysis. Then, the code-to-scenarioscomponent 125 stores the software elements called by each businessprocess in a call graph (also known as a call multigraph), which is adirected graph representing calling relationships between computerprogram procedures. And then, the call graph is utilized to identify thelist of changed scenarios 126. Optionally, the list of changed scenarios126 is provided to the task prioritization component 106, which maycreate task list 130 and/or prioritize the results of the code changeanalysis 124 according to other inputs such as the available parametrictest scenarios 104 and/or the in-use scenarios 116.

In another embodiment, the code-to-scenarios component 125 identifiesthe software elements operated by each business process using a dynamicanalysis. The dynamic analysis may operate on a registration of theprocedures run by the recorded scenarios. This dynamic analysis may alsogenerate some of the in-use scenarios 116.

In still another embodiment, the code-to-scenarios component 125includes code-to-transactions and transaction-to-scenarios. Thisembodiment utilizes an impact analysis module that formulates itsresults in transactions. Optionally, substantially all the scenariosthat utilize the transactions resulted from the impact analysis arepresented in changed scenarios 126.

In one embodiment, a method for comparing old and new customizations inorder to prioritize recorded parametric test scenarios for maintenancepurposes, comprising the following steps: (i) Receiving a manualrecording of a test scenario on an old installation of a data-orientedlarge-scale software system. (ii) Receiving an old customization and anew customization of the data-oriented large-scale software system,wherein the old and new customizations are data structures related tobusiness processes. (iii) Identifying old customization elements thatare different from the new customization elements. (iv) Identifyingchanged customizations that are utilized by the recorded test scenario.And (v) Increasing the priority of fixing the identified changedcustomizations utilized by the recorded test scenario. Optionally, thecustomization changes are changes to SAP Configuration Tables or toOracle Form Personalization.

In one embodiment, a method for prioritizing test plan for customizationchanges according to in-use scenarios, based on comparing old and newcustomizations in order to prioritize recorded parametric test scenariosfor maintenance purposes, including the following steps: (i) Receiving amanual recording of a test scenario on an old installation of adata-oriented large-scale software system. (ii) Receiving an oldcustomization and a new customization of the data-oriented large-scalesoftware system. Optionally, the old and new customizations are datastructures related to business processes. (iii) Identifying oldcustomization elements that are different from the new customizationelements. (iv) Identifying changed customizations utilized by therecorded test scenario. And (V) Increasing the priority of fixing theidentified changed customizations utilized by the recorded testscenario. Optionally, the customization changes are changes to SAPConfiguration Tables and/or to Oracle Form Personalization.

FIG. 1B illustrates a variation of the system illustrated in FIG. 1A,where code changes result from a software upgrade process. Upgradeanalysis component 120, which may run a static impact analysis, outputsa list of code changes 121 resulting from the upgrading process. Thelist of code changes 121 is received by the code-to-scenarios component125 that identifies the list of upgraded scenarios 127, which includesthe business processes affected by the upgrade process.

Optionally, the upgrade analysis component 120 differentiates betweenrequired code changes resulting from the upgrade and required testscenarios resulting from the upgrade. Required code changes resultingfrom the upgrade 121 a should usually be coded before theircorresponding test scenarios can be recorded. The code changes areprovided to a task prioritization component 106 to be added to a tasklist 130. Optionally, after the code changes are implemented, theupgrade analysis analyzes the code again in order to provide an updatedlist of upgraded scenarios 127.

FIG. 1D illustrates one embodiment of sorting impact analysis resultsaccording to the usability of the impacted transactions in a specificorganization. An in-use identifier component 114′ identifiestransactions that are actually in use by the specific organization,referred to as in-use transactions 116′. In some embodiments, the in-useidentifier component 114′ receives inputs from one or more of thefollowing: a usage logger 110′ that registers the used transactions, abug logger 111′ that registers faults in the used transactions, and/orfrom a non-programmer user 100 that identifies the used transactions.The code change analysis component 124 receives as inputs an old codeversion 122 and a new code version 123, and identifies code that waschanged or impacted because of replacing the old code version 122 withthe new code version 123. The identified changed code is received by acode-to-transactions component 125′. Code-to-transactions component 125′identifies the list of changed transactions 126′, which includestransactions that use at least some of the changed code or some of thecode affected by the code change. Task prioritization 106′ sorts thelist of changed transactions 126′ according to the list of in-usetransactions 116′. In one example, the task prioritization 106′ assignsa higher priority to changed transactions that are in-use than tosimilar changed transactions that are not in-use.

FIG. 1E illustrates another embodiment of system components useful forimplementing various embodiments. The incompatible code analyzer 185receives the old vanilla code 180, the new vanilla code 181, andoptionally the release documents 182. The incompatible code is forwardedto the failure point analyzer 186 together with the user-generated code184, in order to generate the code changes to fix 188. The old vanillacode 180, the new vanilla code 181, and the user-generated code 184 arealso received by the transaction analyzer 187, which provides theidentifies transactions to the transaction to code elements component189. The code changes to fix 188 and the code elements from thetransaction to code elements 189 are entered into the transaction-basedtask generator 190, which provides the transaction based task list 191.

Task Prioritization Component 106.

Referring back to FIG. 1A, in some embodiments, a task prioritizationcomponent 106 receives and indexes the parametric test scenarios 104,the in-use scenarios 116, and the scenarios affected by the code changes(referred to as the list of changed scenarios 126). The taskprioritization component 106 outputs a task list 130 to be executed bythe specific organization. The task list 130 describes both “what totest?” (by providing the automatic tests, manual tests, and other tasks)and “how to test?” (by associating the various tests with the properparametric test scenarios 104 that describe how to handle the testresults). The task prioritization component 106 sorts the tests to beexecuted (provided in the task list 130) based on its inputs and logic,such as test a subset of the in-use scenarios affected by an upgrade,and then test a subset of the in-use scenarios unaffected by theupgrade. The “how to test” is available when a parametric test scenario104 covers an in-use scenario, so that running the parametric testscenario 104 can verify the in-use scenario.

In one embodiment, the task prioritization component 106 includes a listof code corrections that are calculated based on inconsistency. Forexample, when the code change analysis component 124 identifies that acertain procedure was deleted because of a code change/upgrade, or thatthe specific organization replaced a certain user-generated code with anon-compatible code, the business processes using the canceled orreplaced procedure are identified as business processes to be fixed.

For clarity, FIG. 1A and some other figures are described in the contextof components and lists. However, various alternative embodiments may beprovided using combined components and/or lists, and/or provided usingdivided and/or additional components and/or lists. In addition, theother figures should cover any kind of modification that divide orintegrate the various activities between the various components, as longas the disclosed system and the modified system provide similar benefitsas a whole.

Identifying non-executable human-readable test scenarios to be updateddue to code changes.

FIG. 2A illustrates one embodiment of a method for identifyingnon-executable human-readable test scenarios to be updated due to codechanges, optionally without utilizing impact analysis. The methodcomprising the following steps: In step 210, receiving an old codeversion and a new code version of a data-oriented large-scale softwaresystem. In step 211, identifying code elements in the old code versionthat were changed in the new code version, referred to as changed code.In step 212, receiving non-executable human-readable test scenarios,each tests at least two transactions. In step 213, analyzing thenon-executable human-readable test scenarios to obtain a list of codeelements utilized by each non-executable human-readable test scenario.Optionally, the non-executable human-readable test scenarios are parsedbefore analyzed. In step 214 running code-to-scenarios analysis toidentify code executed by each non-executable human-readable testscenario. Optionally, an expected failure of each identifiednon-executable human-readable test scenario is described based on thechanged code used by each identified non-executable human-readable testscenario. And in step 215, identifying a subset of the non-executablehuman-readable test scenarios that utilizes the changed code, andtherefore has to be updated. Wherein identifying the subset of thenon-executable human-readable test scenarios based on code-to-scenariosanalysis provides more accurate results than identifying the subsetbased on code-to-transactions analysis. Optionally, the method furthercomprises the step of prioritizing the manual execution of thenon-executable human-readable test scenarios based on the utilization ofthe changed code. Optionally, the non-executable human-readable testscenarios are generated automatically from recordings of useractivities. Optionally, the user is provided with the identifiednon-executable human-readable test scenarios together with the expectedfailures. Optionally, the method further comprises providing a user witha task list for manually executing non-executable human-readable testscenarios, wherein the identified subset of the non-executablehuman-readable test scenarios features higher priorities than theremaining non-executable human-readable test scenarios. And optionally,the changed code comprises changed code customizations.

FIG. 2C illustrates one embodiment of a computer system to identifynon-executable human-readable test scenarios to be updated due to codechanges. The computer system comprising: a storage device 242 containingcomputer usable code, and a processing unit 244 operable to execute thecomputer usable code to: (i) Access an old code version 246 and a newcode version 248 of a data-oriented large-scale software system. (ii)Identify code elements in the old code version 246 that were changed inthe new code version 248, referred to as changed code. (iii) Accessnon-executable human-readable test scenarios 250, each tests at leasttwo transactions. (iv) Analyze the non-executable human-readable testscenarios 250 to obtain a list of code elements utilized by eachnon-executable human-readable test scenario. (v) Run code-to-scenariosanalysis to identify code executed by each non-executable human-readabletest scenario. And (vi) Identify a subset of the non-executablehuman-readable test scenarios that utilizes the changed code, andtherefore has to be updated. Optionally, the processing unit 244 furtherexecutes the computer usable code to prioritize manual execution of thenon-executable human-readable test scenarios 250 based on theutilization of the changed code. Optionally, the processing unit 244further executes the computer usable code to provide a user with theidentified non-executable human-readable test scenarios together withexpected failures. Optionally, the processing unit 244 further executesthe computer usable code to provide a user with a task list for manuallyexecuting non-executable human-readable test scenarios 250, wherein theidentified subset of the non-executable human-readable test scenariosfeatures higher priorities than the remaining non-executablehuman-readable test scenarios. And optionally, the changed codecomprises changed code customizations.

It is to be understood that FIG. 2C can also illustrate many otherdisclosed embodiments that use a storage device containing computerusable code and a processing unit operable to execute the computerusable code. Therefore, for the sake of simplicity and clarity, andwithout any intent to limit the scope of the claims in any way, theapplicants do not provide figures of alternative embodiments that areunderstandable with reference to FIG. 2C.

FIG. 2B illustrates one embodiment of a method for identifyingnon-executable human-readable test scenarios to be updated due to codechanges, utilizing impact analysis. The method comprising the followingsteps. In step 220, receiving an old code version and a new code versionof a data-oriented large-scale software system. In step 221, identifyingchanged code by running impact analysis on the old code version and thenew code version. In step 222, identifying scenarios that use thechanged code, referred to as changed scenarios. Wherein a scenariodefines an activity that operates at least two transactions in aspecific way and therefore scenario-based identification provides moreaccurate results than a transaction-based identification. In step 223,comparing the changed scenarios with non-executable human-readable testscenarios. Optionally, the non-executable human-readable test scenariosare parsed before comparing them with the changed scenarios. And in step224, identifying the non-executable human-readable test scenarios thatcorrespond to the changed scenarios, referred to as error-pronenon-executable human-readable test scenarios. Optionally, the methodfurther comprises the step of prioritizing the manual execution of theidentified non-executable human-readable test scenarios such that theerror-prone non-executable human-readable test scenarios feature higherpriorities than the other non-executable human-readable test scenarios.Optionally, the user is provided with a prioritized task list formanually executing the identified non-executable human-readable testscenarios. Optionally, an expected failure of at least one identifiednon-executable human-readable test scenario is described based on thechanged code used by each identified non-executable human-readable testscenario. Optionally, the user is provided with the identifiednon-executable human-readable test scenarios together with the expectedfailures. And optionally, the changed code comprises changed codecustomizations.

In one embodiment, the task prioritization component 106 identifiesmissing test scenarios 132 when it receives an indication of an in-usescenario 116 that is not covered by the current parametric testscenarios 104. In one embodiment, the missing test scenarios 132 areprovided to the non-programmer user 100 for recording. In anotherembodiment, the missing test scenarios 132 are provided to thenon-programmer user 100 for recording new test scenarios, as part of thetask list 130. If needed, before recording a missing test that is notentirely understandable from the missing tests 132, the non-programmeruser 100 may contact an end-user associated with the missing test toreceive more information.

It is noted that when the recording process is substantiallyunnoticeable or simple enough, even a junior end-user can record testscenarios during his/her regular course of action, or as neededaccording to the list of missing tests 132.

In some embodiments, the task prioritization component 106 may identifyscenarios 126 that utilize recently-updated user-generated code asmissing test scenarios, even when the identified scenarios do notcorrespond to the in-use scenarios 116. The reason is that supportpackages and upgrades by the software platform provider should changeonly the vanilla code and are not directly related to theorganization-specific in-use scenarios. Contrary to changes by thesoftware provider, new changes to the user-generated code are performedon behalf of the specific organization and therefore may indicate thatthe changed code is valuable for the specific organization.

In some embodiments, after a major system upgrade, changed scenarios 126that do not appear in the list of the in-use scenarios 116 areidentified as missing tests bearing lower priority than the changedscenarios 126 that do appear in the list of the in-use scenarios 116.The reason is that these changed scenarios may become important to theorganization as a result of the major system upgrade.

In one embodiment, in-use is detected based on authorizationpreferences, such as Oracle Applications User Management, OracleApplications Responsibility Management, and/or Oracle AuthorizationPolicy Manager. Heuristic analysis (also known as heuristic evaluation)is applied to the various authorizations to reveals the businessprocesses exposed to the non-super-users, which may provide a goodindication of the in-use business processes. Super-users are usuallyexcluded from the heuristic analysis because they may have extrairrelevant authorizations, while non-super-users are usuallyunauthorized to irrelevant operations in order to remove unnecessaryoptions from their menu, improve business agility, manage security froma single place, separates security decisions from application logic, andto enables reuse and sharing of security services.

Automatic identification of missing test scenarios based on a list ofin-use scenarios.

FIG. 3A illustrates one embodiment of a method for identifying missingtest scenarios based on a repository of organization-specific in-usescenarios. The method comprising the following steps: In step 301,accessing a set of scenarios that are actually executed by a specificorganization on a data-oriented large-scale software system, referred toas organization-specific in-use scenarios. Optionally, eachorganization-specific in-use scenarios executes least two transactions.Optionally, the organization-specific in-use scenarios are identified byregistering the executed transactions and registering the order ofexecuting the transactions. Additionally or alternatively, theorganization-specific in-use scenarios are identified by registering thecontext of executing the transactions. Additionally or alternatively,the organization-specific in-use scenarios are identified by registeringthe executed transactions and then converting the registeredtransactions to scenarios utilizing a transaction-to-scenario mapping.In step 302, comparing the organization-specific in-use scenarios withavailable test scenarios repairable by a non-programmer user.Optionally, each available test scenario repairable by a non-programmeruser tests at least two transactions. In step 303, identifyingorganization-specific in-use scenarios that do not have correspondingtest scenarios repairable by a non-programmer user, referred to as firstset of missing test scenarios. And in step 304, providing at least oneof the missing test scenarios to a non-programmer user for generating atest scenario repairable by a non-programmer user, which covers the atleast one missing test scenario. Optionally, the step of generating thetest scenario repairable by a non-programmer user comprises recordingthe non-programmer user executing the missing test scenario. Optionally,the generated test scenario repairable by a non-programmer is aparametric test scenario repairable by a non-programmer user, whichdefines different alternatives for executing the test scenario.Optionally, the parametric test scenario repairable by a non-programmeruser further defines different alternatives for evaluating the executionoutcome, and is expressed in terms that substantially conform to termsused in the non-programmer user's work environment, and coherent withactual manual operation of the test scenario. Optionally, the methodfurther comprises the steps of identifying a second set of missing testscenarios having scenarios that utilize recently-updated user-generatedcode; assigning the first set of missing test scenarios a higherpriority than the second set of missing test scenarios; and providingthe first and second sets to the non-programmer user for generatingcorresponding parametric test scenario repairable by a non-programmeruser. And optionally, the data-oriented large-scale software system hada recent major system upgrade, and further comprising the steps ofidentifying most of the scenarios that were changed because of thesystem upgrade as missing test scenarios; and assigning higher priorityto changed scenarios that do have corresponding organization-specificin-use scenarios than to changed scenarios that do not havecorresponding organization-specific in-use scenarios; whereby certainscenarios may become important to the organization as a result of themajor system upgrade.

FIG. 3B illustrates one embodiment of a computer system to identifymissing test scenarios based on a repository of organization-specificin-use scenarios. The computer system comprising: a storage device 312containing computer usable code, and a processing unit 314 operable toexecute the computer usable code to: (i) Access a set of scenarios thatare actually executed by a specific organization on a data-orientedlarge-scale software system, referred to as organization-specific in-usescenarios 316. Optionally, Each organization-specific in-use scenarioexecutes least two transactions. (ii) Compare the organization-specificin-use scenarios 316 with available test scenarios repairable by anon-programmer user 318. Optionally, each available test scenariorepairable by a non-programmer user tests at least two transactions.(iii) Identify organization-specific in-use scenarios that do not havecorresponding test scenarios repairable by a non-programmer user,referred to as first set of missing test scenarios. And (iv) Provide atleast one of the missing test scenarios to a non-programmer user 100 togenerate a test scenario repairable by a non-programmer user 320 thatcovers the at least one missing test scenario. Optionally, theprocessing unit further executes the computer usable code to identifythe organization-specific in-use scenarios by registering the executedtransactions and registering the order of executing the transactions.Optionally, the processing unit further executes the computer usablecode to identify the organization-specific in-use scenarios byregistering context of executing the transactions. Optionally, theprocessing unit further executes the computer usable code to identify asecond set of missing test scenarios comprising scenarios that utilizerecently-updated user-generated code, assign the first set of missingtest scenarios a higher priority than the second set of missing testscenarios, and provide the first and second sets to the non-programmeruser to generate the corresponding parametric test scenarios repairableby a non-programmer user. And optionally, the data-oriented large-scalesoftware system had a recent major system upgrade, and the processingunit further executes the computer usable code to: identify most of thescenarios that were changed because of the system upgrade as missingtest scenarios, and assign higher priority to changed scenarios that dohave corresponding organization-specific in-use scenarios than tochanged scenarios that do not have corresponding organization-specificin-use scenarios.

FIG. 3C illustrates one embodiment of a method for identifying missingtest scenarios based on authorization policies. The method comprisingthe following steps: In step 350, analyzing the authorization policiesapplied to non-super-users of a specific data-oriented large-scalesoftware system in order to identify organization-specific in-usescenarios. The reason for analyzing the authorization policies appliedto non-super-users is that the non-super-users are usually unauthorizedto irrelevant operations. In step 352, comparing the identifiedorganization-specific in-use scenarios with available test scenariosrepairable by a non-programmer user. Optionally, each available testscenario repairable by a non-programmer user tests at least twotransactions. In step 354, identifying organization-specific in-usescenarios that do not have corresponding test scenarios repairable by anon-programmer user, referred to as first set of missing test scenarios.And in step 356, providing at least one of the missing test scenarios toa non-programmer user, having proper authorizations, for generating atest scenario repairable by a non-programmer user that covers the atleast one missing test scenario. Optionally, the data-orientedlarge-scale software system is Oracle ERP and the authorization policiesare taken from Oracle Applications User Management and/or from OracleApplications Responsibility Management. Optionally, the generated testscenario repairable by a non-programmer is a parametric test scenariorepairable by a non-programmer user, which defines differentalternatives for executing the test scenario. Optionally, the parametrictest scenario repairable by a non-programmer user further definesdifferent alternatives for evaluating the execution outcome, and isexpressed in terms that substantially conform to terms used in thenon-programmer user's work environment, and coherent with actual manualoperation of the test scenario.

In one embodiment, the data-oriented large-scale software system isOracle ERP. Oracle Applications User Management and Oracle ApplicationsResponsibility Management are used to assess the authorizations becauseOracle Applications Responsibility Management includes the menus exposedto the users, and the exposed menus set the screens that are used by theusers. Therefore, the forms exposed to each user are identified byanalyzing the responsibilities. Optionally, Oracle transactions areselected from the group of forms, workflow, reports, concurrentprograms, and other executable units. Optionally, Oracle businessprocesses are selected from the group of forms, workflows, reports,and/or concurrent programs.

FIG. 3D illustrates one embodiment of a computer system to identifymissing test scenarios based on authorization policies. The computersystem comprising: a storage device 362 containing computer usable code,and a processing unit 364 operable to execute the computer usable codeto: (i) analyze the authorization policies 366 applied tonon-super-users of a specific data-oriented large-scale software systemin order to identify organization-specific in-use scenarios. (ii)Compare the identified organization-specific in-use scenarios withavailable test scenarios repairable by a non-programmer user 368, whereeach available test scenario repairable by a non-programmer user testsat least two transactions. (iii) Identify organization-specific in-usescenarios that do not have corresponding test scenarios repairable by anon-programmer user, referred to as first set of missing test scenarios.And (iv) provide at least one of the missing test scenarios to anon-programmer user 100, having proper authorizations, to generate atest scenario repairable by a non-programmer user 370 that covers the atleast one missing test scenario. Optionally, the data-orientedlarge-scale software system is Oracle ERP and the authorization policiesare taken from Oracle Applications User Management and OracleApplications Responsibility Management. Optionally, the generated testscenario repairable by a non-programmer is a parametric test scenariorepairable by a non-programmer user, which defines differentalternatives for executing the test scenario. And optionally, theparametric test scenario repairable by a non-programmer user furtherdefines different alternatives for evaluating the execution outcome, andis expressed in terms that substantially conform to terms used in thenon-programmer user's work environment, and coherent with actual manualoperation of the test scenario.

Automatic Identification of Missing Test Scenarios Based on AnalyzingTest Results of a Changed Business Process.

In one embodiment, a method for identifying missing test scenarios basedon changes in test results comprising the following steps: (i) Receivingan indication of a code change or a customization change applied to abusiness process. (ii) Executing substantially all parametric testscenarios related to the business process. And (iii) if all theparametric test scenarios were executed properly, suggesting that theexecuted parametric test scenarios are insufficient to test the changedbusiness process.

In another embodiment, a method for identifying missing test scenariosbased on changes in test results comprising the following steps: (i)Receiving the results of executing essentially all parametric testscenarios related to a business process before making a code orcustomization change to the business process. (ii) Receiving the resultsof executing essentially all parametric test scenarios related to thebusiness process after making the code or customization change to thebusiness process. (iii) Searching for behavioral changes reflected inthe differences between the results received before and after making thechanges to the business process. And (iv) Suggesting that the executedparametric test scenarios are insufficient to test the business processif less than a predetermined amount of behavioral changes are found.

Automatic Testing Component 150.

In some embodiments, the task list 130 is executed automatically by anautomatic testing component 150. Optionally, the automatic testingcomponent 150 provides a completion report 152 that lists thesuccessfully completed tests, and a need to fix report 154 that liststhe parametric test scenarios 104 that have to be fixed. The testscenarios on the need to fix report 154 are forwarded to thenon-programmer user 100 to be re-recorded based on their descriptions.

In one embodiment, the automatic testing component 150 runs on an imageof the tested system. The image is updated from time to time in order toensure that the tests run on relevant data. Optionally, the automatictesting component 150 queries a database of the new installation inorder to obtain relevant values to be entered. One example of queryingthe database includes the following steps: reading from a parametrictest scenarios 104 a parametric entry value representing a catalognumber of a certain product available in a certain warehouse; theautomatic testing component 150 querying the database of the newinstallation for one or more catalog numbers of the certain product inthe certain warehouse; substituting the returned value into theparametric entry value; and trying to execute the test scenario.Optionally, the automatic testing component 150 retrieves multiplevalues from the database and tries executing the test scenario multipletimes with different values. In addition, the automatic testingcomponent 150 may calculate the proper value of a parametric entryvalue. For example, the automatic testing component 150 may calculate afuture date or multiply a result by a predefined factor.

Running automatic tests on a data-oriented large-scale software system,such as ERP, is a complicated task because the ERP environment is fullof dependencies and preconditions making it impossible to runprerecorded test scenarios with their originally recorded values. As aresult, the automatic testing component 150 has to substitute validvalues into the executed test, where the valid values should fit thespecific installation and state of data. For example, a user records ascenario that opens a new order, ships a product from a warehouse, andthen closes the order. However, the next day when the automatic testingcomponent 150 will try to rerun the recorded scenario it will have tosubstitute a different order number (because the recorded order numberalready exists in the system), will have to ship a different unit of theproduct from the warehouse (because the recorded product was shippedand, thus, does not exist anymore in the warehouse), and then will haveto use the new order number to close the order. Therefore, the obsoleterecorded values have to be replaced by valid values.

The automatic testing component 150 also has to handle system, setup,and/or business process changes, which occur from time to time. Forexample, after recording a test scenario, a new unrecorded field may bemarked as mandatory. Thus, the resulting parametric test scenario 104has to be completed, manually or automatically, with the newly addedmandatory field.

Failures to execute parametric test scenarios 104, also referred to asbugs, may result from both code errors and obsolete parametric testscenarios 104. Code errors may be fixed by code changes. Obsoleteparametric test scenarios 104 may result from either incompleteparametric test scenarios 104 or inaccurate parametric test scenarios104. Incomplete parametric test scenarios 104 do not include clearspecification of how to handle certain instances of a business process,optionally resulting from generating the parametric test scenarios 104from partial data, such as a set of recordings that do not cover all thevariations of the business process. Inaccurate parametric test scenarios104 do not describe the current version of the business process well,optionally resulting from updating the business process without updatingits related parametric test scenarios 104. The obsolete parametric testscenarios 104 may be fixed by updating the parametric test scenarios104, optionally using re-recordings, wizards, automatic updates based onsimilar updates, and/or manual updates of the test specifications.

Optionally (not illustrated in FIG. 1A), the need to fix report 154includes both bugs resulting from obsolete parametric test scenarios 104and bugs resulting from code errors. The bugs resulting from code errorsmay be added to the list of code changes 121 in order to be fixed bycode changes, and then, as described above, the code changes areassociated with the affected scenarios that are to be retested.

In one embodiment, all the tests detailed in the need to fix report 154,which are the tests that were not completed successfully by theautomatic testing component 150, are forwarded to the non-programmeruser 100 who decides which of the tests have to be re-recorded and whichof the tests have to be forwarded to the programmer for code change. Inthis embodiment, the non-programmer user 100 analyzes the tests on theneed to fix report 154 manually because there are cases where theautomatic testing component 150 fails while the code is fine. Forexample, the automatic testing component 150 may fail executing aparametric test scenario 104 by opening order that is not in delayinstead of a order in delay (as needed by the specific scenario). As aresult, the automatic testing will not receive the screen that dealswith the delay, and the execution will fail. In such examples, thenon-programmer user 100 may re-record the scenario in a way that willenable the test recording and analysis component 102 to understand ithas to select an open order in delay, and/or add proper comments to theparametric test scenario 104. For example, in order to indicate that thescenario has to receive an open order, the user may add to there-recording a step of querying the system about the opened scenarios.

Alternatively, the non-programmer user 100 may update the scenario usinga wizard that helps in parameterizing the scenario. In some embodiments,a business process may change to a degree where it is better to replacethe previous parametric test scenario 104 with a new parametric testscenario 104, created from a newly recorded test scenario, andpreferably recorded on the new installation.

In one embodiment, the automatic testing component 150 determinesautomatically, based on the type of failure, which of the unsuccessfultests are to be forwarded to the non-programmer user 100, and which areto be forwarded to a programmer. For example, a parametric test scenario104 failing with an internal system error, an activation error, or acompilation error—is forwarded to a programmer for code change. Examplesof internal system errors include exceptions in Java and unexpectederrors in ABAP. Optionally, the remaining failures, such as failed testscenarios that do not result in the expected values, are forwarded tothe non-programmer user 100.

In one embodiment, a test scenario that failed when executed by theautomatic testing component 150 is executed automatically until itsfailure point, or until a few steps ahead of its failure point. Then,the non-programmer user 100 fixes the test scenario by re-recording thetest scenario using the GUI. And then the system updates the parametrictest scenario 104 automatically based on the initial semiautomatic runand the later manual recording.

Optionally, and assuming only an intermediate section of the parametrictest scenario is identified as different in the re-recording of the testscenario (compared to the failed test scenario), the system checkswhether it can continue executing the remaining of the failed testscenario. If it can continue, the user is not required to re-record theremaining steps of the failed test scenario.

In one embodiment, the user is directed to re-record a scenario in a waythat will enable the test recording and analysis component 102 toidentify the right parameter for a certain screen. For example, in orderto indicate the test recording and analysis component 102 that ascenario has to receive an open order, the user may re-record anadditional step of querying the system about the opened scenarios.Alternatively, the non-programmer user 100 may update the scenario usinga wizard that helps in parameterizing the scenario. In some embodiments,a business process may change to a degree where it is better to replacethe previous parametric test scenario 104 with a new parametric testscenario 104, created from a newly recorded test scenario, andpreferably recorded on the new installation.

Methods for Handling Failed Test Scenarios.

FIG. 6A illustrates one embodiment of a method for determining whetherto correct a failed test scenario using re-recording or code change. Themethod comprising the following steps. In step 610, receivingdescriptions of test scenarios that failed while executed by anautomatic testing component; wherein at least 50% of the test scenariosare expressed by user interaction activities and not code-relatedlanguage. In step 612, identifying the test scenarios that failed withabnormal termination as test scenarios to be forwarded to a programmerfor code change, wherein abnormal termination does not include normaltermination with a value different from an expected value. Optionally,the abnormal termination is selected from the group of: internal systemerrors, activation errors, and compilation errors. Optionally, thecomputer runs Java programming language, and the internal system errorsare Java exceptions. Alternatively, the computer runs ABAP programminglanguage, and the internal system errors are ABAP unexpected errors. Andin step 614, identifying the test scenarios that failed without abnormaltermination as test scenarios to be forwarded to a non-programmer userfor re-recording or manual evaluation. In some cases, re-recording ofscenarios that failed with abnormal termination has less than 20% chanceto fix the abnormal termination failure, and therefore the testscenarios that failed with abnormal termination are forwarded to theprogrammer for code change. Optionally, the automatic testing componentexecutes the test scenarios on a data-oriented large-scale softwaresystem, and the test scenarios that failed without abnormal terminationcomprise test scenarios that do not result in predefined expected valuesbecause of insufficient description of the test scenarios, while thedata-oriented large-scale software system's code is expected to beerror-free; and generating a human-readable notice to requestconfirmation to add execution result to expected results of the testscenario. Optionally, the method further comprises identifying a testscenario that failed without abnormal termination because of a missingprerequisite, and generating a human-readable notice to encourage thenon-programmer user to re-record one or more additional steps to revealthe missing prerequisites. Optionally, the method further comprisesidentifying a test scenario that failed without abnormal terminationbecause of a missing prerequisite, and generating a human-readablenotice to suggest a possible value for the missing prerequisite.Optionally, the method further comprises identifying a test scenariothat failed without abnormal termination because of a missingprerequisite, and directing the non-programmer user to re-record ascenario in a way that enables a test recording and analysis componentto identify a functioning parameter for a certain screen. Optionally,the manual evaluation of the test scenarios that failed without abnormaltermination comprise activities selected from the group of: indicatingthat the test scenarios are in order although resulted in errormessages, changing preferences indicating that the received messages areacceptable, adding expected results to approve the test scenarios,and/or re-recording the test scenarios

FIG. 6B illustrates one embodiment of a method for processing failuresof executing test scenarios over a data-oriented large-scale softwaresystem. The method comprising the following steps. In step 620,identifying failures of the test scenarios due to logical errors in thefailed test scenarios. Optionally, the logical errors result fromparametric test scenarios that do not include clear specification of howto handle certain instances of a business process. Optionally, thelogical errors result from missing coverage of all the variations of thebusiness process tested by the test scenario. Optionally, the logicalerrors result from an inaccurate description of the business process. Instep 622, forwarding the identified logical related failed testscenarios to a non-programmer user for re-recording. Optionally, thenon-programmer user is directed to re-record a scenario in a way thatenables a test recording and analysis component to identify a missingprerequisite for a certain screen. In step 624, identifying failures ofthe test scenarios due to code errors. Optionally, the code errors causeabnormal termination, and the abnormal termination does not includenormal termination with a value different from expected value.Optionally, the abnormal termination is selected from the group of:internal system errors, activation errors, and compilation errors.Optionally, the computer runs Java programming language, and theinternal system errors are Java exceptions. Optionally, the computerruns ABAP programming language, and the internal system errors are ABAPunexpected errors. And in step 626, forwarding the identified coderelated failed test scenarios to a programmer for code change.

FIG. 6C illustrates one embodiment of a method for correcting testscenarios using re-recording. The method comprising the following steps:In step 630, receiving descriptions of test scenarios that failed whileexecuted by an automatic testing component; wherein at least 50% of thetest scenarios are expressed by user interaction activities and notcode-related language. In step 632, identifying the test scenarios thatfailed due to insufficient description while the code is fine.Optionally, the step of identifying the test scenarios that failed dueto the insufficient description is performed by a non-programmer user.In step 634, identifying the test scenarios that failed due to codeerror. Optionally, the step of identifying the test scenarios thatfailed due to code error is performed by a non-programmer user. In step636, re-recording, by a non-programmer user on a data-orientedlarge-scale software system, the test scenarios that failed due to theinsufficient description while the code is fine. And in step 638,forwarding to a programmer for code change the test scenarios thatfailed due to code error. Optionally, a non-programmer user identifiestest scenarios that failed with an abnormal termination as testscenarios that failed due to code error, wherein abnormal terminationdoes not include normal termination with a value different from anexpected value. Optionally, a non-programmer user identifies testscenarios that do not result in predefined expected values as testscenarios that were failed due to the insufficient description while thecode is fine.

FIG. 6D illustrates one embodiment of a method for automaticauthorization-based routing of failed test scenarios. The methodcomprising the following steps: In step 640, receiving descriptions oftest scenarios that failed while executed by an automatic testingcomponent on a data-oriented large-scale software system; wherein atleast 50% of the test scenarios are expressed by user interactionactivities and not code-related language. In step 642, identifying thetest scenarios that failed with abnormal termination as test scenariosto be forwarded to a first programmer authorized to change thedata-oriented large-scale software system; wherein abnormal terminationdoes not include normal termination with a value different from anexpected value. And in step 644, identifying the test scenarios thatfailed without abnormal termination as test scenarios to be forwarded toa second programmer unauthorized to change the data-oriented large-scalesoftware system.

In one embodiment, the system tries to execute available parametric testscenarios 104 automatically. If the system succeeds, it provides acompletion report 152 and exists. If the execution of the parametrictest scenarios 104 fails, the system sends to the non-programmer user100 a message describing the test scenarios that have to be run manuallyand/or re-recorded (referred to as need to fix report 154). If thenon-programmer user 100 decides that the parametric test scenario 104failed due to a code error, the parametric test scenario 104 isforwarded to a programmer for fixing, and then the system triesexecuting the fixed parametric test scenario 104 again. Optionally theprocess repeats until the parametric test scenario 104 runssuccessfully. If the non-programmer user 100 decides that the parametrictest scenario 104 failed due to a business process change, and/or due topoor representation of the test scenario, the non-programmer user 100updates the parametric test scenario 104 by re-recording the testscenario. Then, the system tries executing the updated parametric testscenario 104 again. Optionally, the process repeats until the parametrictest scenario 104 runs successfully.

The parametric test scenarios 104 may be executed periodically, andespecially when the organization-specific in-use software elements aremodified or upgraded. The automatic testing component 150 may test thesystem seamlessly, and request a user intervention only uponunsuccessful execution of a parametric test scenario 104, optionallyresulting from a bug or a business process change.

FIG. 1C illustrates a variation of the system illustrated in FIG. 1A,where the parametric test scenarios 104 are executed manually 150′. Testlist 130′ describes the testing program for the manual execution,completion report 152′ describes the successfully completed tests, andneed to fix report 154′ describes the failed tests.

Test Recording and Analysis Component 102.

In some embodiments, a non-programmer user 100 records a test scenarioon an old installation, and the test recording and analysis component102 converts the recording to a parametric test scenario 104 able to beexecuted on the new installation. In some embodiments, the parametrictest scenario 104 comprises two corresponding documents. The first is aparametric test scenario repairable by a non-programmer user, whichenables the non-programmer user 100 to execute and re-record the testscenario manually. The second is a coded parametric test scenario, whichis used by the automatic testing component 150 for automatic executionof the parametric test scenario 104.

FIG. 4 illustrates one embodiment of a record and play process. Ascenario is recorded on an old installation 420. The recording reflectsthe old code 422 (including old vanilla code and old user-generatedcode) and the old database 424 (including old data and oldcustomizations). Because the code of the new installation 432 and thedatabase of the new installation 434 are different from the code of theold installation 422 and the database of the old installation 424, therecorded scenario is parameterized 426 into a parametric test scenario104 comprising parametric entries, which describe the values to beentered, and parametric expected values, which describe the predictedresults. Then, the parametric test scenario 104 is executed on the newinstallation 430 by a program that interacts with the code of the newinstallation 432 (including new vanilla code and new user-generatedcode) and the database of the new installation 434 (including new dataand new customizations), which may be quite different from the code anddatabase of the old installation.

Referring again to the definition of “parametric test scenariorepairable by a non-programmer user”, in some embodiments, some of theparametric test scenarios 104 would fail when played on an upgradedrelease and/or on other installations. And because the test scenariopseudocode is written with expressions that substantially conform to theuser's work environment, the non-programmer user is expected to be ableto repair many of the failures manually. The ability to repair thefailures provides a great advantage over cases where the non-programmeruser does not understand the parametric test scenario documentation, andtherefore cannot usually repair the test scenario.

In one embodiment, the automatically generated parametric test scenariorepairable by a non-programmer user provides clear distinction betweenvalues to be entered, field labels, and procedure calls in order toimprove its readability. In another embodiment, the automaticallygenerated parametric test scenario repairable by a non-programmer userdistinguishes between entry values and expected values in order to siftout irrelevant data from the test scenario document. For example, theuser selects a customer (entry value), and then the system pops up thedefault parameters of that customer (expected values). In thisembodiment, the parametric test scenario repairable by a non-programmeruser guides the user as to which values to enter, and may inform theuser of the expected values resulting from the entered value. On theother hand, a parametric test scenario that is irreparable by anon-programmer user (such as an application showing mere screen shots)would show the user both the entry values and the expected values withno distinction.

The ability to execute, on the new installation, recordings of testscenarios taken on the old installation, provides a great unobviousadvantage at least because (i) the non-programmer user 100 is wellfamiliar with the old installation, while may not be familiar with thenew installation; (ii) the old installation is assumed to operate well,while the customizations made to the new installation are assumed tohave bugs; and (iii) some of the originally recorded values do not fitnew installation.

In addition, the ability to generate automatically the parametric testscenarios 104 from recordings provides a significant advantage due tosome or all of the following benefits: the scenarios can be recorded byusers, such as business analysts or business process experts, having noprogramming/engineering partners; there is almost no need for technicalskills because recording a scenario does not require scripting,parameterization, or modularization; and there is no need to documentthe recording manually when the system automatically generates humanreadable parametric test scenarios from the recordings.

In one example, the test recording component 102 records one or more ofthe following data while the user normally runs a business process:executed transactions, sequence of screen inputs, issued messages,outputs to user interface, database activities, and/or objects involvedin a certain transaction.

In some embodiments, the ability to generate a parametric test scenario104 automatically, and specifically a parametric test scenariorepairable by a non-programmer user, is based on dynamic analysis,understanding of the work environment, and static analysis.

The dynamic analysis traces and analyzes the script produced in run-timeby the recording tool. Optionally, the script produced in run-time issent to a designated server not in the premises of the end-user.

Understanding of the work environment enables automatic generation of aparametric test scenario repairable by a non-programmer user that is:(a) expressed in terms that substantially conform to the terms used inthe user's work environment, and (b) coherent with the actual manualoperation of the test scenario. The result is a user-friendly documentthat, for example, includes the field labels that the user is used towork with, requires the user to enter the entry values, but at leastmost of the times does not require the user to enter the expectedvalues.

Static analysis (also known as “impact analysis”) identifies the logicalentities associated with the entered values, identifies therelationships between the logical entities, identifies the relationshipsand dependencies between the different business processes, andidentifies the relationships between the different software elements.These identified relationships are used to convert values in therecorded test scenarios to variables. And because the variables canreceive values other than the recorded values, the parametric testscenario may still be able to run on a different installation, and/orassist the user in repairing the test scenario. For example, twoenvironmental changes that may affect the execution of a previouslyrecorded test scenario include a data change (such as when the originalinput is currently invalid), and/or operation over differentinstallations. In one numerical example, a user records a scenario witha customer whose ID is 1234. After upgrading the system to a newerrelease, customer 1234 is no longer available, and thus the recordedtest scenario cannot run as is. However, the static analysis identifiesthe value 1234 as a customer ID number and selects a different value forthe customer ID number, for example by extracting an alternative validcustomer number from a database, guessing another customer ID, or tryingsubstituting customer ID used in other scenarios. A similar exampleincludes a change to an element name/property, etc.

In one embodiment, a method for automatic generation of test scenarioswith expected behavior from recordings, comprising the following steps:(i) Recording a user entering a test scenario, wherein most of the useractivities required for the recording are substantially the same asend-user activities. (ii) Analyzing the recording using dynamic analysisand domain knowledge. (iii) Inferring expected results from therecording and domain knowledge, optionally using static analysis. And(iv) Generating from the recording a test scenario repairable by anon-programmer user.

Improving the Ability to Execute, Manually or Automatically, aParametric Test Scenario 104.

The ability to execute a parametric test scenario 104 is usually limitedto the logic and type of data exposed by the information used to createthe parametric test scenario 104. Various recordings of the samescenario may provide additional information necessary for certaininstances of the scenario. Various recordings of the same test scenariomay be combined into a single parametric test scenario 104, or saved asa plurality of parametric test scenarios 104, each useful in a differentinstance of the scenario. For example, assuming a first recording of acertain test scenario is taken by a high-priority user who is notrequired to go through an authorization phase. As a result, execution ofthe resulting parametric test scenario 104, by a low-priority user, mayfail because the authorization phase is undocumented. Consequently, asecond recording of the certain test scenario, taken by a low-priorityuser who goes through the authorization phase, will provide theinformation necessary to document the instance of executing the testscenario by the low-priority user.

In one embodiment, the various test scenarios are stored essentiallyindependently and a functioning test scenario is selected and executedbased on the available conditions. For example, if a low-priority userexecutes the scenario, the parametric test scenario 104 generated fromthe second recording should be executed. Alternatively, all the variousrecordings are executed until one of them succeeds.

In another embodiment, information describing the various instances ofthe scenario, gathered from the various recordings, is integrated into amaster parametric test scenario 104. Optionally, the integration isbased on identifying the common denominator between the variousrecordings. Additionally or alternatively, the integration is based onintegrating the different instances to a conditioned branched testscenario able to deal with variations that were not explicitly recorded.

In some embodiments, the master parametric test scenario 104 alsoincludes instances that were not explicitly recorded and/or explicitlyentered to the system. These instances are derived from combinations ofthe various instances. For example, a first recorded scenario includingscreen series (a, b, c, d, e) and a second recorded scenario includingscreen series (a, f, c, g, e) are analyzed. The analysis result may bethat the screen series (a, b, c, g, e), which was not explicitlyrecorded, may also be legal because either screen d or g can followscreen c.

In some embodiments, a new installation may modify the logic of abusiness process associated with a parametric test scenario 104, and asa result, the execution of that parametric test scenario 104 may fail.In one embodiment, the non-programmer user 100 repairs the parametrictest scenario 104 by re-recording the test scenario with the relevantchanges that reflect the new process. The new recording is then used toupdate the old parametric test scenario, optionally automatically.

Optionally, there are a few related parametric test scenarios 104incompatible with the new installation. In this case, when anon-programmer user 100 records a modification to a previously recordedparametric test scenario 104, the automatic testing component 150 checkswhether a similar modification can be applied to the other relatedincompatible parametric test scenarios 104. Parametric test scenarios104 are identified as related when including overlapping portions abovea threshold. For example, more than 80% overlap in the screen flow, ormore than 95% overlap in the order of accessing vanilla code procedures.The modifications may also be applied to unrelated parametric testscenarios as long as the modified sections are located in essentiallyequivalent subsections. Then, optionally, the system tries executing themodified parametric test scenarios 104 to verify correctness.

For example, a few scenarios related to the same business process wererecorded on an old installation. After a system upgrade, the scenariosfailed because of a newly added screen. However, the newly added screenmay be common to at least some of the scenarios, thus enabling thesystem to apply substantially the same modifications to other scenarioshaving similar screen flows. In another example, a few scenarios, whichwere recorded on an old installation, share a common object. Aftersystem upgrade, the scenarios failed because the common object includesa new field. Therefore, after one of the transactions is fixed, e.g. byentering a value to the new field, the system tries applyingsubstantially the same modification to the common object in the othertransactions, e.g. by entering the same type of value to the new field.

In some embodiments, the automatic testing component 150 tries handingfailures by drawing lots or making smart guesses based on the type ofthe new or modified elements, known conditions, or known relationships.For example, after system upgrade, a scenario fails because it includesa new field. So the automatic testing component 150 tries substitutingdifferent values retrieved from the database based on relationshipsbetween the new field and the database (for example, a drop-down list isconnected to a database table); based on relationships between the newfield and other known fields (for example, new field X should receivethe value of field Y times a certain gain); and/or based on the type ofthe new field's data, such as a date. In another example, proper valuesare guessed and the test scenario is potentially executed until one ofthe tests results in a successful run.

In some embodiments, each parametric test scenario 104 comprises twocorresponding documents (i) a parametric test scenario repairable by anon-programmer user, for example to be used by the non-programmer user100; and (ii) a corresponding coded parametric test scenario, forexample to be used by the automatic testing component 150 for runningthe parametric test scenarios 104 on different installations and/or withdifferent data. Examples of various logics that enable automaticexecution of the parametric test scenarios 104 are describedhereinbelow.

(a) The parametric test scenarios repairable by a non-programmer userand the corresponding coded parametric test scenarios may includedifferent data. For example, for each scenario, the parametric testscenario repairable by a non-programmer user may include the transactionnumber and screen number only once, while the coded parametric testscenario may include duplications of that data. The duplications mayassist queries such as a query designed to ascertain which specificsteps in the test scenarios are to be affected by a change to a specificscreen in a specific transaction. The parametric test scenariorepairable by a non-programmer user may also include comments in naturallanguage that do not appear in the corresponding coded parametric testscenario.

(b) Time dependency of a field, such as a future value, a past value, ora specific time interval.

(c) Back-off value assignment logic that tries substituting alternativevalues in case of failure. For example, the automatic testing component150 tries substituting a first value from a list of suppliers into aparameter describing a supplier in a parametric test scenario 104. Ifthe test scenario fails, the automatic testing component 150 triessubstituting an alternative supplier value from the list into theparameter in the parametric test scenario 104 and then tries running theparametric test scenario 104 again, optionally until one of theexecutions succeeds or a predetermined number of attempts is reached. Inanother example, the automatic testing component 150 tries substitutinga first future date into a parameter describing a future date in theparametric test scenario 104. If the test scenario fails, the automatictesting component 150 tries substituting an alternative future date andthen tries running the parametric test scenario 104 again, optionallyuntil one of the executions succeeds or a predetermined number ofattempts is reached.

(d) In some embodiments, dynamic analysis and/or static analysisdetermine the values to be substituted into the parameters of theparametric test scenario 104 based on preconditions. The preconditionsmay be inferred from the recordings and/or from the transaction codeitself. Examples for inferring preconditions by identifyingrelationships between software elements using dynamic analysis include:(i) a user runs a query and then enters the result into a field; thedynamic analysis identifies the relationship between the two softwareelements; identifies the table in the database from which the value isextracted; and set the field to receive a value selected from thattable. (ii) a user enters a value into a field; the dynamic analysisidentifies a relationship between the entered value and a correlatedsoftware elements of the same type that is associated with the sametransaction, such as a future date (associated with the date ofexecuting the transaction), or payment with commission (associated withan invoice on the same currency issued a few steps behind); identifiesthe origin of the correlated software element, such as a certain tablein the database, or the result of a certain calculation; and set thefield to receive a value selected from that table or calculation. Otherexamples for inferring preconditions by analyzing the transaction orbusiness process code, optionally using static analysis, include: (i)inferring the precondition from a calculation made on a required field.For example, when the static analysis determines that the transactioncode checks whether a certain filed value represents a future date or anatural number, it is reasonable to infer that the certain field shouldreceive a future date or a natural number. And (ii) when the staticanalysis determines that the transaction code verifies that a certainfiled value appears on a certain table in the database, it is reasonableto infer that the certain field should receive a value selected from thecertain table.

The automatic testing component 150 also has to analyze the executionoutcome in order to determine its successfulness. In one embodiment, theautomatic testing component 150 is based on the following three keyissues: (i) Selecting valid input values. (ii) Overcoming run-timeobstacles, such as warning messages and small changes in the systembehavior resulting from changes in the data and/or system setup. And(iii) Determining execution result based on the significant outcomes.Some of the outcomes, such as the type of a previous screen, the type ofa completion message, or the value of specific fields are moresignificant for determining the execution successfulness than otheroutcomes, such as the task's completion time and date, or the operatorname.

In some embodiments, new test scenarios are generated as variations offunctioning test scenarios that are executed semiautomatically. The useof functioning test scenarios as the basis of generating a new testscenario improves the work efficiency significantly and enables thenon-programmer user to focus on the unique aspects of the new testscenario instead of rerecording known steps. The non-programmer user maychange the semiautomatically entered data, may add new data to the basicfunctioning test scenario, may add new forms (also known as screens) tothe basic functioning test scenario, and/or enter missing data. Theuser's interactions with the semiautomatic executed test scenario arerecorded to generate the new test scenario.

The ability to add new forms to the basic functioning test scenarios issignificantly important because it enables the non-programmer user torecord new test scenarios that are not limited to trivial variations ofthe basic functioning test scenario. Using functioning test scenariosalso provides a great advantage over variations generated, by the way,based on failed test scenarios. This is because when dealing with failedtest scenarios, the non-programmer user is focused on fixing the failedtest scenario, and usually the new test scenario will replace the failedtest scenario. Contrary to failed test scenarios, selecting afunctioning test scenario as the baseline improves the non-programmeruser's confidence (because the functioning test scenario should operatewell) and focuses the non-programmer user on generating new additionaltest scenarios, instead of fixing the available failed test scenario.

Generating a new test scenario based on executing a functioning testscenario semiautomatically.

FIG. 7 illustrates one embodiment of a method for semiautomaticexecution of a functioning test scenario to enable a non-programmer userto generate a new test scenario based on the functioning test scenario.The method comprising the following steps. In step 702, recording thenon-programmer user while interacting with the semiautomatic executionof the functioning test scenario. And in step 704, generating the newtest scenario based on the recording. Wherein the semiautomaticexecution of the functioning test scenario enables the followingfeatures via the following steps. In step 706, automatically enteringdata inferred from the functioning test scenario into a form of adata-oriented large-scale software system. In step 708, enabling thenon-programmer user to change the automatically entered data andregistering at least one change to the automatically entered data. Instep 710, enabling the non-programmer user to add a new form to thesemiautomatically executed functioning test scenario, and registering atleast one new added form. And in step 712, enabling the non-programmeruser to enter data into fields inside the new form, and registering thedata entered to at least one field inside the new form. Optionally, themethod further comprises the step of comparing forms following the newform with available forms of the functioning test scenario, andautomatically entering data inferred from the functioning test scenariointo the following forms when there is a match. Optionally, the methodfurther comprises the step of using dynamic analysis and domainknowledge for generating the new test scenario. Optionally, the methodfurther comprises the step of using static analysis to infer expectedresults when generating the new test scenario. Optionally, thefunctioning test scenario and the new test scenario are test scenariosrepairable by a non-programmer user. Optionally, the test scenariosrepairable by a non-programmer user are parametric test scenariosrepairable by a non-programmer user. Optionally, the new test scenarioconsists of the recorded forms. Optionally, the method further comprisesproviding a wizard to assist the non-programmer user in changing theautomatically entered data. Optionally, the new test scenario isgenerated based on data gathered by the wizard. Optionally, the methodfurther comprises enabling the non-programmer user to change theautomatically entered data includes enabling the non-programmer user tocomplete missing data. Optionally, the semiautomatic execution of thefunctioning test scenario further marks automatically the fields havingthe missing data.

FIG. 8A, FIG. 8B, and FIG. 8C illustrate certain aspects of the methodfor semiautomatic execution of a functioning test scenario to enable anon-programmer user to generate a new test scenario based on thefunctioning test scenario, as described above in relation to FIG. 7.

Referring to FIG. 8A, screen 810 illustrates a semiautomatic executionof the functioning test scenario. The product model “55655” was inferredfrom the functioning test scenario and entered automatically into theform of the data-oriented large-scale software system. The semiautomaticprogram code enables the non-programmer user to change the automaticallyentered data, as illustrated in FIG. 8B, where the previous value “5565”was changed by the user to “44444”. FIG. 8B also illustrated how theuser added the credit card number “12345678”, which is recorded and willbe added to the new test scenario to be generated based on therecording.

FIG. 8C illustrates the ability to add a new form to thesemiautomatically executed functioning test scenario. The functioningtest scenario included only screens 820, 824, and 828. While using theoriginal semiautomatic execution, and the user added screens 825 and826, and thus the new test scenario that was generated based on thefunctioning test scenario now includes the two additional screens 825and 826. The user may add data to the new screens in a similar manner tothe credit card example of FIG. 8B.

In one embodiment, a method for updating a test scenario repairable by anon-programmer user based on semiautomatic execution of the testscenario, includes the following steps: (i) executing semiautomaticallythe test scenario repairable by a non-programmer user by automaticallyentering data inferred from the test scenario to a form. (ii) enabling auser to amend the automatically entered data. (iii) using, by anautomatic test scenario generation engine, the user amendments togenerate an updated test scenario repairable by a non-programmer. (iv)counting the number of user amendments to a certain field acrossdifferent execution of the test scenario. And (v) after the usersuccessively amends the certain field for predefined times (greater thantwo, and preferably greater than five), marking the certain field, bythe automatic test scenario generation engine, as a known-to-be-wrongfield. Optionally, the method further includes requesting a user toenter data manually to the known-to-be-wrong field. Optionally, themethod further includes forwarding the known-to-be-wrong field to aprogrammer, whereby the programmer replaces the known-to-be-wrong fieldwith a functioning parametric representation. Optionally, the predefinedtimes for successively amending the certain field is at least 5 times;alternatively, the predefined times are at least ten times over at leastten successive executions of the test scenario.

Some scenarios may include optional screens and/or optional fields (forsimplicity usually referred to as optional screens), which may or maynot appear depending on various conditions. Examples of optional screensinclude authorization screen to approve a financial transaction above acertain threshold, additional signatures for a transaction executed by ajunior user, various warnings of expected events or minor mismatches,and optional suggested supplements or sales depending on variouspreconditions. In one embodiment, the non-programmer user 100 identifiesthe optional screens while recording the test scenario using a wizard orby performing a predefined activity, such as using the keyboard, mouseor other data entry device, or combination thereof. In one example, thecontrol button and right mouse click are used to select optional fields.In another embodiment, the optional screens are marked with a predefinedtext string on the parametric test scenario 104. In another embodiment,the recorded fields and screens are displayed to the non-programmer user100, who identifies the mandatory screens and fields, or alternativelyidentifies the optional screens and fields.

FIG. 5A, FIG. 5B, and FIG. 5C illustrate three instances of the samescenario, which result in different screen flows. FIG. 5A illustratesthe basic screens (510, 511, and 512). FIG. 5B illustrates the basicscreens (510, 511, and 512), and optional screen 514. And FIG. 5Cillustrates the basic screens (510, 511, and 512), and optional screen516. FIG. 5D illustrates a master scenario including the variousidentified screens, where screens 514 and 516 are marked as optionalscreens. In one embodiment, a draft of the master scenario, and/ordifferent instances of the scenario, are provided to a non-programmeruser 100 for manual identification of the optional screens 514 and 516.In another embodiment, different instances of the same scenario areanalyzed automatically or semi-automatically to identify the optionalscreens 514 and 516.

In some embodiments, parametric test scenarios 104 are generated fromrecordings of key-users while reducing the interactions with thekey-users to a minimum, or even without notifying the key-users of thefact that their activities are recorded to generate parametric testscenarios 104.

One method for minimizing user interactions when recording scenarios forgenerating parametric test scenarios 104 includes the following steps:(i) Defining one or more key-users who supposed to operate the businessprocesses to be tested. (ii) Recording the key-users while working,optionally essentially without affecting their usual course of work.(iii) Generating automatically parametric test scenarios 104 from therecordings. Optionally, the alternative instances of the same testscenario are aggregated into a master parametric test scenarios 104having branches describing the different instances. (iv) Prioritizingthe generated parametric test scenarios 104 based on the list of changedscenarios 126. And (v) executing the generated parametric test scenarios104 on a different installation, according to the calculatedprioritization, without bothering the key-users.

The method above enables the key-users to supply most of the informationneeded to create the parametric test scenarios 104. Optionally, thegenerated parametric test scenarios 104 are executed on a differentinstallation by an automatic testing component 150. Optionally, failedparametric test scenarios 104 are forwarded to a non-programmer user100, who is not the key-user that recorded the scenarios used togenerate the failed parametric test scenarios 104. Moreover, thegenerated parametric test scenarios 104 may be executed on the differentinstallation by a quality assurance (QA) user who is less familiar withthe system relative to the key-user that recorded the scenarios used togenerate the parametric test scenarios 104.

Improving user friendliness of a manual test scenario generated fromrecordings by removing unessential test steps.

In one embodiment, improving the user friendliness of a manual testscenario is expressed by reducing the number of fields described in themanual test scenario. Additionally or alternatively, improving the userfriendliness of a manual test scenario may also be expressed in reducingthe number of fields described in the manual test scenario.

FIG. 9 illustrates one embodiment of a method for improving userfriendliness of a non-executable human-readable test scenario byremoving unessential test steps. The method comprising the followingsteps. In step 910, receiving a non-executable human-readable testscenario comprising unessential test steps. Optionally, the receivednon-executable human-readable test scenario was generated from arecording, and thus includes unessential test steps. In step 912,identifying the unessential test steps. And in step 914, generating auser friendlier non-executable human-readable test scenario by removingthe identified unessential test steps. Optionally, a screen that doesnot return a value usable for its calling screen is identified as anunessential screen that generates unessential testing step. Optionally,a screen that can be replaced by its returned value without changing themain outcome of the test scenario is identified as an unessentialtesting step. Optionally, the text describing the steps related to thescreen that can be replaced by the returned value is replaced with arequest to insert the returned value. Optionally, the unessential teststeps comprise a screen that returns a value that the user alreadyshould have known. Optionally, the unessential test steps comprise entryvalues identical to the default values. Optionally, all explanationsrelated to default values are essentially removed from the userfriendlier non-executable human-readable test scenario. Optionally, theunessential test steps comprise unproductive steps, which result in anerror message, whereby a following successful variation of theunproductive steps is left in the user friendlier non-executablehuman-readable test scenario.

FIG. 10A illustrates one embodiment of a method for improving the userfriendliness of a non-executable human-readable test scenario, byremoving unessential test steps. The method comprising the followingsteps. In step 940, receiving a non-executable human-readable testscenario comprising unessential test steps. Optionally, the receivednon-executable human-readable test scenario was generated from arecording. In step 942, generating a graph of screens that are referredto by the non-executable human-readable test scenario. In step 944,identifying a side branch that is not mandatory for the proper operationof the non-executable human-readable test scenario. Optionally, thenon-mandatory side branch comprises a help screen. Optionally, thenon-mandatory side branch comprises unproductive steps resulted in anerror message. Optionally, the non-mandatory side branch provides avalue identical to a default value. Optionally, the non-mandatory sidebranch is characterized by not changing the behavior of a businessprocesses under test. And in step 946, removing the description of thenon-mandatory side branch from the non-executable human-readable testscenario.

FIG. 10B illustrates the graph of screens already discussed above inFIG. 10A. The non-executable human-readable test scenario originallyincluded screens 950, 952, 954, 956, and 958. However, screen 945 wasidentified as a side branch that is not mandatory for proper operationof the non-executable human-readable test scenario. Therefore, thedescription of the dashed line is removed from the non-executablehuman-readable test scenario.

FIG. 11 illustrates one embodiment of a method for user-friendly testscenario recording. The method comprising the following steps. In step970, enabling a non-programmer user to record a test scenario on adata-oriented large-scale software system, while using help screen, andwhile running a sanity check that is not mandatory to the recorded testscenario and does not change the test scenario's outcome. In step 972,identifying steps associated with using the help screen and running thesanity check. And in step 974, generating a non-executablehuman-readable test scenario from the recording without the identifiedsteps. Optionally, the step of generating a non-executablehuman-readable test scenario from the recording is executed before thestep of identifying the steps associated with using the help screen andrunning the sanity check, and the identified steps are removed from thegenerated non-executable human-readable test scenario. Optionally, themethod further comprises the steps of enabling the user, whilerecording, to exploit a screen that can be replaced by its returnedvalue without changing main outcome of the test scenario, and generatingthe non-executable human-readable test scenario without the replaceablescreen. Optionally, the method further comprises the steps of enablingthe user, while recording, to enter values identical to default values,and generating the non-executable human-readable test scenario withoutthe steps of entering the values identical to the default values.

In some embodiments, the parametric test scenarios 104 include a layeredstructure, and the process of updating an existing parametric testscenario 104 maintains these layers. Saving the parametric testscenarios 104 with layers enables the system to update the parametrictest scenarios 104, while preserving the properties of at least some ofthe parameters, without having to run dynamic and static analyses inorder to understand what each parameter is and what rules it obeys. Forexample, based on a parametric layer indicating that field X shouldreceive a future date or that the value of field X is a function of thevalue of field Y, the automatic testing component 150 can enter to fieldX a current future date or the value of field Y without having to runstatic or dynamic analysis. Moreover, the updated test scenario mayinherit the properties of the older test scenario, such as theimportance of the test scenario and the list of the scenario's users.

Non-limiting examples of layers include: (i) Comments, such as commentsby a programmer which do not appear on the transaction screen. (ii)Field properties, which may be identified from previous analyses, suchas a future date, current date, version of the tested software release,and/or version of the test scenario. The field properties enable theautomatic testing component 150 and/or a human tester to select theright value to be substituted into the field even when the parametrictest scenario 104 is executed on an installation different from theinstallation on which it was recorded. (iii) Transaction and screennumber relative to each text block (for cases where each scenario isrelated to a transaction, and each transaction has multiple screens). Inone embodiment, the transaction number appears in multiple places in theparametric test scenario 104, and after a change, all the otheroccurrences are updated automatically. In one embodiment, the layer ofthe transaction and screen numbers for each text block are utilized tolocate all the tests affected by a screen/transaction change. Forexample, a change is made, impact analysis identifies the numbers of theaffected transactions and screens, then these numbers are located at theparametric test scenario 104, and a user is asked to fix the specificlocations in order to be able to run the tests properly.

In some embodiments, additional information required for executing aparametric test scenario 104 is obtained using a combination of scenariorecordings followed by focused wizard-aided queries, according to thefollowing method steps: (i) Recording a scenario, optionally by anon-programmer user 100. (ii) Generating a parametric test scenario 104from the recorded scenario. Alternatively, the parametric test scenario104 is generated by a test engineer or by a wizard-based software. (iii)Executing the parametric test scenario 104 manually or by the automatictesting component 150. And (iv) Upon encountering an execution failure,fixing the failed test scenario by receiving from a human operatoradditional information related to the failure using a wizard.

When possible, the wizard suggests the human operator the possiblevalues, dependencies, preconditions, preferences, or logic for eachrelevant field or activity, and the human operator indicates thesuitable description aimed to solve the execution failure. Optionally,the information related to the failure is received from a non-programmeruser 100 that defines at least the problematic fields, for example, byindicating a valid subset of values suiting the specific instance of thetest scenario, by indicating a dependency, and/or by indicating therelevant ratio or calculation to achieve the right value.

The combination of generating a parametric test scenario 104 from therecorded scenario and then adding additional information using thewizard simplifies the process of generating a complex parametric testscenario 104 because the user uses the wizard mostly to add informationrelated to the failure and/or complicated parts, and not for creatingthe entire parametric test scenario 104. In one example, the wizard isused to gather less than 20% of the total information available in theparametric test scenario 104. And optionally, the remaining at least 80%of the information is derived from one or more recordings of the testscenario, which did not involve the wizard.

In another example, the end-user activities are recorded, optionallywithout informing the end-user of the recording. Then the test recordingand analysis component 102 generates parametric test scenarios 104.Optionally, the generated parametric test scenarios 104 are executed bythe automatic testing component 150, and the need to fix tests 154 areprovided to a non-programmer user 100 that enters the missinginformation using a wizard. In one embodiment, the data extracted fromthe recordings is presented by the wizard as the default value. Thisexample shortens the time to complete the wizard because plain recordingrequires significantly less time than interacting with the wizard.

Formulating Results of Code Change Analysis According to in-UseScenarios.

In one embodiment, a method for prioritizing a task plan comprisingresults of code change analysis according to organization-specificin-use scenarios, comprising the following steps: (i) Receiving an oldcode version and a new code version of a non organization-specificdata-oriented large-scale software system. (ii) Running code changeanalysis to identify code elements in the old code version that werechanged in the new code, referred to as changed code. (iii) Runningcode-to-scenarios analysis to identify scenarios that use the changedcode, referred to as changed scenarios. Wherein the code-to-scenariosanalysis also analyzes scenarios that are not in-use by theorganization, and therefore some of the scenarios identified by thecode-to-scenarios analysis are not in-use by the organization. (iv)Prioritizing a test plan for the changed scenarios based on theorganization-specific in-use scenarios, such that changed scenarios thatare identified as in-use scenarios receive a higher priority. Whereinthe organization-specific in-use scenarios refer to scenarios that areactually executed by the specific organization. Optionally, the testplan prioritization is proportional to the amount of code changesrelevant to each scenario. And optionally, the test plan prioritizationis proportional to the weight of the corresponding in-use scenarios.

In one embodiment, at least some of the scenarios analyzed by the impactanalysis are received from the manufacturer documentation and/or basedon data gathered from other organizations. In another embodiment, theimpacted scenarios are achieved by one or more of the following methods:(i) Using a scenario to code mapping, which may be received from runtime logging that registers the used code elements in different level,such as transactions, procedures, and code lines. (ii) Using scenario totransaction mapping, and then transaction to code mapping, wherein thecode related to the transactions is compared with the impacted codeidentified by the impact analysis.

Generating Task Plan for Fixing and Testing User-Generated CodeAccording to Changes in Non Organization-Specific Code.

In one embodiment, a method for prioritizing a task plan comprisingfixing user-generated code tasks according to organization-specificin-use scenarios, comprising the following steps: (i) Receiving an oldcode version and a new code version of a non organization-specificdata-oriented large-scale software system. (ii) Running code changeanalysis to identify code elements in the old code version that werechanged in the new code, referred to as changed code. (iii) Running codeanalysis to identify user-generated code that needs to be fixed to suitthe new code version. (iv) Running code-to-scenarios analysis toidentify scenarios that use the user-generated code to be fixed,referred to as impacted scenarios. (v) Accessing a list oforganization-specific in-use scenarios. (vi) identifying theintersection of the impacted scenarios and the organization-specificin-use scenarios. And (vii) generating a task plan that focuses a testeron testing the identified intersection by mainly presenting the fixingprerequisites for testing the identified intersection. Optionally, thetask plan includes tests for various impacted scenarios having differentprerequisites, and the prerequisites are used to indicate essentiallythe earliest time to test each impacted scenario without having topotentially repeat the test. Optionally, the task plan prioritization isproportional to the amount of code changes relevant to each scenario.Optionally, the task plan prioritization is proportional to the weightof the corresponding in-use scenarios. Optionally, the task plancomprises high priority tasks and low priority tasks, and mainlypresenting the fixing prerequisites for testing the identifiedintersection means that more than half of the high priority tasks relateto the identified intersection. And optionally, focusing the tester ontesting the identified intersection means that most of the highestpriority tasks relate to the identified intersection.

Similar to the above method, the following method formulates results ofcode change analysis according to organization-specific in-usetransactions, instead of scenarios. The method comprising the followingsteps: (i) Receiving an old code version and a new code version of adata-oriented large-scale software system. (ii) Running code changeanalysis to identify code element in the old code version that werechanged in the new code, referred to as changed code. Optionally, thechanged code comprises the procedures that include the code that isdifferent between the two versions. Optionally, the changed codecomprises code impacted by replacing the old code version with the newcode version. (iii) Running code-to-transactions analysis to identifytransactions that use the changed code, referred to as impactedtransactions. Wherein the code-to-transactions analysis also analyzestransactions that are not in-use by the organization, and therefore someof the transactions identified by the code-to-transactions analysis arenot in-use by the organization. (iv) Comparing the organization-specificin-use transactions with the impacted transactions to identifyorganization-specific in-use transactions that utilize at least some ofthe impacted code. Optionally, the transactions that utilize theimpacted code are transactions that can execute the impacted code on theold code version according to a certain run-time analysis. Optionally,the transactions that utilize the impacted code are transactions thatcan contain the impacted code according to static analysis of the oldcode version. Optionally, the organization-specific in-use transactionsrefer to transactions that are actually in-use by the specificorganization. And (v) increasing prioritization assigned to changed codethat is utilized by the organization-specific in-use transactions.

According to another embodiment, a method for sorting changed codeaccording to organization-specific in-use scenarios, and without usingimpact analysis, comprising the following steps: (i) Receiving an oldcode version and a new code version of a data-oriented large-scalesoftware system. (ii) Identifying code element in the old code versionthat were changed in the new code, referred to as changed code. (iii)Identifying the changed code that is executed by theorganization-specific in-use scenarios. Wherein theorganization-specific in-use scenarios refer to scenarios that areactually in-use by the specific organization. And (iv) Marking thechanged code that is executed by the organization-specific in-usescenarios as more error-prone than changed code that is not executed bythe organization-specific in-use scenarios.

According to still another embodiment, a method for formulating resultsof code change analysis according to available parametric testscenarios, directly without using intermediate changed scenarios,comprising the following steps: (i) Receiving an old code version and anew code version. (ii) Identifying code impacted by replacing the oldcode version with the new code version. (iii) Analyzing availableparametric test scenarios for identifying the parametric test scenariosthat utilize at least some of the impacted code. Wherein the parametrictest scenarios describe how to execute test scenarios and how toevaluate the execution outcomes. And (iv) Increasing prioritizationassigned to impacted code that is utilized by the parametric testscenarios.

Risk Assessment.

In order to estimate software reliability from test data, test caseshave to resemble the operational usage of the software. Allocation ofthe test scenarios plays an important role in estimating softwarereliability, where the better the allocation of test scenarios, thebetter the level of input coverage. Test scenarios based on operationalusage are sometimes known as operation-based testing, which is a type ofusage-based statistical testing.

Software needs to be tested extensively before it is considereddependable and trustworthy. To guide testing, software developers oftenuse an operational profile, which is a quantitative representation ofhow a system is used. One way of ensuring that the most used functionsof a system are tested is achieved by documenting user inputs and theiroccurrence probabilities in a given operational profile. Test cases canbe generated directly from such an operational profile. Operationalprofiles are also a necessary part of quality-of-service predictionmethods for software architectures, because these models have to includeuser inputs into their calculations.

In some embodiments, methods for risk assessment are designed to reducethe time and resources required to test software updates and upgrades.The risk assessment methods focus the tests on routines having higherprobability to generate bugs, and optionally assist in creating softwaretest scenarios to test these routines. In some embodiments, the higherprobability to generate bugs results from adding user-generated code toand/or customizing parts of a data-oriented large-scale software system,such as SAP ERP or Oracle ERP, to the needs of a specific organization.In these embodiments, the method may prompt the user to test mainly theorganization-specific in-use software elements, which are the relevantsoftware elements for the specific organization. In other embodiments,the data-oriented large-scale software system is upgraded, thereforerequiring the specific organization to re-test its use of the complexsoftware platform along with its user-generated code.

FIG. 12 illustrates one embodiment of a method for directing the usersas to what software modules need to be checked when facing a codechange, such as an upgrade from an old installation to a newerinstallation, using the following steps.

In step 210 r, identifying scenarios in-use by a specific organization.The in-use property may be binary (e.g. in-use or not in-use), or may beweighted according to the scenario's importance (e.g., not in-usescenario, low importance in-use scenario, medium importance in-usescenario, and high importance in-use scenario). In one embodiment, thescenarios in-use by the specific organization are identified based onregistration of prior system usage on the old installation of thespecific organization. Step 210 r also identifies theorganization-specific in-use software elements. In another embodiment,the scenarios in-use by the specific organization are estimated based onusage of equivalent scenarios by similar organizations, where similarorganizations include organizations in the same field of operation,organizations running similar mixture of business-processes, and/ororganizations having similar customization data structures that definethe business processes.

In one example, a method for identifying scenarios in-use by a firstorganization based on similarity, comprising the following steps:identifying a second organization similar to the first organization;deciding which of the parametric test scenarios that are in-use by thesimilar organization are relevant to the specific organization; andlooking for equivalent parametric test scenarios. Optionally, the firstand second organizations are in the same field of operation. Optionally,the overlapping of the business-processes run by the first and secondorganizations is more than 80%.

In step 212 r, identifying business processes covered by parametric testscenarios available for the old installation. Optionally, step 212 rassumes the available parametric test scenarios cover scenariosimportant to the specific organization. Step 212 r also identifies theorganization-specific in-use software elements that are tested by theavailable parametric test scenarios. In one embodiment, the availableparametric test scenarios for the old installation are parametric testscenarios repairable by a non-programmer user.

In step 214 r identifying on the new installation (after the codechange/software upgrade), by a component equivalent to code-to-scenariocomponent 125, the organization-specific in-use software elements usedby the in-use scenarios of step 210 r and the tested-processes of step212 r. The modules may be identified by comparing the old and newinstallations. In one embodiment, the comparison mainly includescomparing the names of the vanilla code procedures, because an upgradedsoftware release usually maintains the procedure names of its previousrelease. In another embodiment, the comparison compares the screen flowsand when the overlapping passes a threshold—the scenarios are identifiedas corresponding scenarios. In still another embodiment, the comparisonincludes a control flow analysis that receives entry point andidentifies the accessed vanilla code elements using techniques such asdead code analysis or dynamic resolving.

In step 216 r, assessing failure probability of theorganization-specific in-use software elements from step 214 r. In oneembodiment, it is assumed that (i) software elements withoutuser-generated code would not produce bugs, and (ii) software elementswith user-generated code would produce bugs.

In some embodiments, a software element (which may be for example aprogram object, a procedure, a class, or a library) is considered toinclude a user-generated code when it has been changed or had one ormore attributes altered.

In another embodiment, a software upgrade process is analyzed toidentify the software elements affected by the upgrade process and assesthe failure probability of the affected software elements, using thefollowing assumptions: (i) software elements unchanged by the upgradeprocess, which do not include user-generated code, would not producebugs; (ii) software elements changed by the upgrade process, which donot include user-generated code, have a small probability to producebugs; (iii) software elements unchanged by the upgrade process, whichinclude user-generated code, have a significant probability to producebugs; and (iv) software elements changed by the upgrade process, whichinclude user-generated code, have a high probability to produce bugs.

In some embodiments, the software elements affected by the upgradeprocess are identified using impact analysis, also known as static codeanalysis. Then, static code analysis is used to identify thetransactions and/or business processes using the affected softwareelements.

In still another embodiment, the failure probability of a businessprocess is determined based on the existence or inexistence ofpreviously reported bugs in the business process by users of thespecific organization and/or by users from other organizations runningthe same business process. Optionally, the more bugs reported for abusiness process, the higher its failure probability. The previouslyreported bugs may be extracted from a failure-reporting logger, from aquality center, or from any other historical data logger. Optionally,this embodiment assumes that software elements changed by the upgradeprocess, which do not include user-generated code, may have asignificant probability to produce bugs. In one example, after anupgrade it is assumed that all code, including unchanged vanilla code issusceptible to bugs.

In still another embodiment, the failure probability of a businessprocess is determined based on its dependency on other businessprocesses. For example, the more business processes depending on aspecific business process, the higher the failure probability of thespecific business process. Optionally, this embodiment also assumes thatsoftware elements changed by the upgrade process, which do not includeuser-generated code, may have a significant probability to produce bugs.

In one embodiment, the risk of each scenario is calculated by summingthe failure probabilities of the software elements it operates on thenew installation. Alternatively, the risk is binary and the scenario isconsidered to have a high risk if one of more of its software elementsis estimated to have a high failure probability; the scenario isconsidered to have a low risk if none of its software elements isestimated to have high failure probability. Alternatively, the risk iscalculated based on the number and/or type of users that use thescenarios, and the risk associated with frequently used scenarios isincreased.

FIG. 13 is a schematic illustration of a failure analysis component 170,which may include a bug analysis component 172 that registers open andclosed bugs, a change analysis component 174 similar to the code changeanalysis component 124, a customization analysis component 176 based onstatic impact analysis on the code that read the customization and/ordynamic analysis of recordings, and a document analysis component 178.In some embodiments, the output of the failure analysis component 170also includes the list of changed scenarios 126.

FIG. 14 is a schematic illustration of the likelihood of failure andrisk assessment components. Risk assessment 252 receives the in-usescenarios 256, likelihood of failure 251, and user specification 260.The risk assessment 252 is then used for generating the prioritized testplan 254. In-use scenarios 256 describe the scenarios and/or codeelements that are actually in-use by the analyzed organization. Userspecification 260 provides the user preferences for building the testplan, such as ranking of the various operators and ranking theimportance of the changed scenarios, and/or gross decisions such asdetermining that all vanilla code changed by an upgrade process shouldbe checked, or all vanilla code changed by an upgrade process is assumedto operate well. Changed scenarios 258 receive inputs from an impactanalysis code change 270, from a document analysis 272, fromcustomization change 274, and from user specification 260.

Likelihood of failure 251 receives the changes scenarios 258, old bugs268, results from automatic testing 262, in-use elements from recordings276, code metrics 264, and/or data from cross organizations 266. In oneembodiment, the cross organizations 266 provides list of scenarios andtheir execution results. In one embodiment, the in-use elements fromrecordings 276 are obtained by dynamic analysis of test scenariorecordings that identify the code elements used in the runtime. In oneembodiment, results from automatic testing 262 for a portion of theparametric test scenarios may assist in determining the likelihood offailure of the tasks, such as code changes and testing updates, relatedto the automatically executed parametric test scenarios. For example,failure in executing a parametric test scenario by the automatic testing262 may increase the likelihood of failure assigned to the otherparametric test scenarios related to the same business process, whilesuccess in executing a parametric test scenario by the automatic testing262 may decrease the likelihood of failure assigned to the otherparametric test scenarios related to the same business process.

Referring back to FIG. 12, in step 218 r, based on the previous steps,the test scenarios related to the organization-specific in-use softwareelements (i.e. the test scenarios that are relevant to the specificorganization) are sorted into the following four categories, which areused for generating a task list 130:

(i) Important scenarios that are at high risk to be damaged because ofthe code change, such as a system upgrade, for which no parametric testscenarios 104 exist. For this group the non-programmer user 100 may beprompted to record parametric test scenarios 104 during upgradepreparations.

(ii) Important scenarios that are at low risk to be damaged because ofthe upgrade, for which no parametric test scenarios 104 exist. For thisgroup the non-programmer user 100 may be prompted to record theparametric test scenarios 104 during upgrade preparations only when timepermits.

In one embodiment, the immediately above categories (i) and (ii) arecalculated using the following steps: (a) obtaining the relativecomplement of the in-use scenarios (from step 210 r) with respect to thescenarios covered by the available parametric test scenarios (from step212 r) on the old installation (referred to as the uncovered scenarios).In other words, obtaining the set of scenarios included in the in-usedscenarios 116 and missing from the parametric test scenarios 104; (b)identifying on the new installation the software elements used by theuncovered scenarios (using step 214 r); then (c) estimating the risks ofeach uncovered scenario based on summing the products of the softwareelements it uses on the new installation and their failure probabilities(from step 216 r). The resulting high-risk scenarios belong to category(i), while the resulting low-risk scenarios belong to category (ii).

(iii) Important scenarios that are at high risk to be damaged because ofthe upgrade, for which parametric test scenarios 104 do exist. For thisgroup the non-programmer user 100 may be prompted to run the relevantparametric test scenarios 104 during upgrade.

(iv) Important scenarios that are at low risk to be damaged because ofthe upgrade, for which parametric test scenarios 104 do exist. For thisgroup the non-programmer user 100 may be prompted to run the relevantparametric test scenarios 104 only when time permits.

In one embodiment, the immediately above categories (iii) and (iv) arecalculated using the following steps: (a) identifying on the newinstallation the software elements used by the scenarios covered by theavailable parametric test scenarios 104 (from step 212 r); then (b)estimating the risks of each scenario covered by the availableparametric test scenarios based on summing the product of the softwareelements it uses on the new installation and their failure probabilities(from step 216 r). The resulting high-risk scenarios belong to category(iii), while the resulting low-risk scenarios belong to category (iv).

In some embodiments, steps 210 r and/or 218 r are customized accordingto a required profile.

In one example, step 210 r is modified to easily set the testingpriorities by limiting the first group of in-use scenarios to scenariosused by specific users, specific departments, and/or specific roles inthe specific organization. As a numeric example, a finance departmenthas 100 users. 5 managers are identified as most critical to theorganization, 10 others are critical, and the rest are important but notcritical. Therefore, only the activities of the 5 most critical, or and15 most critical and critical managers may be logged. Alternatively, themost extensive data logging and performance analysis is applied to the 5most critical managers, then some less extensive analysis is applied tothe 10 critical managers, and the least extensive analysis is applied tothe rest of the users. In still another example, it is determined that afirst group of department is more important to the organization than asecond group of departments. Thus, a most extensive analysis is appliedto the activities of the more important departments.

In another example, step 210 r includes business processes that werepreviously reported as having bugs by users of the specific organization(may be extracted from a failure reporting logger of the organization),and step 218 r displays the potentially error-prone software elementsversus the upgrade changes. Note that this example customizes the reportaccording to the past bugs in the specific organization, becausedifferent organizations usually have different user-generated code, andtherefore are susceptible to different bugs.

In another example, step 218 r is customized by adjusting theorganization-specific in-use software elements to include only one ormore of the following: actually-in-use software elements, manuallyidentified software elements, software elements related to thebusiness-critical processes, software elements used by specific user/s,software elements used by a specific department, and/or potentiallyerror-prone software elements. In a modification of the last numericalexample, steps 210 r to 216 r are executed as described. Then, for thepurpose of obtaining the relevant tests to be executed, theorganization-specific in-use software elements are limited to (i) onlythe 5 most critical managers, (ii) only software elements associatedwith senior users in, or the higher the user's location at theorganization hierarchical structure—the greater his associatedimportance weight, and/or (iii) just specific departments. The systemthen presents the testing categories of step 218 r according to the 5most critical managers. In an alternative example, the system presentsthe testing categories of step 218 r according to a subset of apredefined business-critical processes (although the information forother less critical processes is available, it is not presented).

FIG. 15A illustrates one embodiment where steps 210 r and 216 r areutilized to display the test scenarios according to their importance(extracted from step 210 r) and failure probability (extracted from step216 r). Test scenario 391 features low importance and low failureprobability, test scenario 392 features high importance and low failureprobability, test scenarios 393 and 394 feature low importance and highfailure probability, and test scenario 395 features high importance andhigh failure probability. The sorting of the illustrated test scenariosmay help in prioritizing the test plan.

The disclosed embodiments may be used for displaying the test scenariosaccording to various axis, and are not limited in any way to importanceversus usability. For example, FIG. 15B illustrates one embodiment wherethe test scenarios are displayed according to importance of eachscenario versus number of users of the scenarios. FIG. 15C illustratesone embodiment where the test scenarios are displayed according toimportance of each scenario versus the type of the test scenarios (e.g.manual executed parametric test scenario or automatically executedparametric test scenario). FIG. 15D illustrates one embodiment where thetest scenarios are displayed according to the modules to which the testscenarios relate versus the estimated complexity to adapt the varioustest scenarios to the upgrade.

Cross-Organization Reuse.

In some embodiments, a user belonging to a certain organization accessesa catalog of parametric test scenarios 104, which were created based ondata from other organizations. The user automatically or semiautomatically adapts the required parametric test scenarios 104 to itsspecific installation using the following steps: locating theappropriate parametric test scenarios 104 in the catalog; and adaptingthe parametric test scenarios 104 to the certain organization.

Software as a Service (SaaS) enables easy cross-organization sharing ofavailable tests. In one embodiment, a non-programmer user in oneorganization records a scenario. Then the scenario is converted to aparametric test scenario 104 and offered to users of otherorganizations, which may run it on the web without having to installanything. In another embodiment, a user in a certain organization cansee a catalog of parametric test scenarios 104 generated from recordingsof users in other organizations. The parametric test scenarios 104 areautomatically adapted to the specific configuration of the certainorganization, although they were recorded on other installations. Instill another embodiment, an online parametric test catalog is adaptedautomatically to a specific organization using the following steps:accessing a parametric test catalog created from recordings of otherorganizations; and when a user in a certain organization wants to searchthe catalog for relevant parametric test scenarios 104, the systemautomatically substitutes suitable values to the parameters according tothe needs of the certain organization, so that the non-programmer user100 will understand what each scenario tests (because it is adapted tohis familiar work environment). Optionally, the parametric testscenarios 104 are shown to the non-programmer user 100 according to theold installation, and then automatically updated to the specification ofthe new installation.

In some embodiments, similar organizations that can potentially exchangedata useful for automatic scenario correction and/or automatic codecorrection share similar usage logging 110, and share similarcustomizations, such as similar SAP Configuration Tables or similarOracle Form Personalization.

In one embodiment, a method for reporting failures before executionbased on results from an equivalent organization includes the followingsteps: locating substantially equivalent parametric test scenariosbetween the current organization and a reference organization thatunderwent substantially the same software upgrade to a data-orientedlarge-scale software system; and before executing the substantiallyequivalent parametric test scenarios on the current organization,reporting expected failures in the substantially equivalent parametrictest scenarios, based on the failures observed in the referenceorganization. Optionally, at least some of the corrections made to fixthe substantially equivalent parametric test scenarios on the referenceorganization are suggested to the current organization. Optionally, morethan half of the corrections made to fix the substantially equivalentparametric test scenarios on the reference organization are suggested tothe current organization. Optionally, the parametric test scenarios areexecuted manually, and at least some of the reported expected failuresare fixed before executing the substantially equivalent parametric testscenarios on the updated installation of the current organization.Optionally, more than half of the reported expected failures are fixedbefore executing the substantially equivalent parametric test scenarioson the updated installation of the current organization. Optionally, thesubstantially equivalent parametric test scenarios call the sameApplication Program Interface (API). Optionally, the referenceorganization changed the API in order to adapt for the upgrade, and anequivalent change is suggested to the current organization. Optionally,the substantially equivalent parametric test scenarios refer to batchprograms that operate the same vanilla programs.

In one embodiment, a method for reusing code corrections includes thefollowing steps: receiving vanilla software elements and screen flow ofa first parametric test scenario executed by a first organization facingan upgrade from a first to a second version of a data-orientedlarge-scale software system; identifying a second organization thatunderwent substantially the same upgrade from the first to the secondversion of the data-oriented large-scale software system, and had asecond parametric test scenario that executed substantially the samevanilla software elements and screen flow as the first parametric testscenario; obtaining the results of executing the second parametric testscenario on the second data-oriented large-scale software system of thesecond organization before adjusting the second parametric test scenarioto the upgrade; and estimating, before executing the first parametrictest scenario on the second data-oriented large-scale software system ofthe first organization, whether the first parametric test scenario willsucceed or fail based on the obtained results of the second parametrictest scenario. Optionally, obtaining the changes made by the secondorganization to adjust the second parametric test scenario to theupgrade; and suggesting the first organization to implement equivalentchanges to adjust the first parametric test scenario to the upgrade.Optionally, implement automatically the equivalent changes to adjust thefirst parametric test scenario to the upgrade. Optionally, implement theequivalent changes to adjust the first parametric test scenario to theupgrade and then manually executing, on the first time, the firstparametric test scenario on the second version of the data-orientedlarge-scale software system. Optionally, adjusting the second parametrictest scenario to the upgrade comprises updating the second parametrictest scenario. Optionally, adjusting the second parametric test scenarioto the upgrade comprises fixing a user-generated code. Optionally,adjusting the second parametric test scenario to the upgrade comprisesreporting a code error in the vanilla software.

In one embodiment, a method for detecting failures without executingparametric test scenarios includes the following steps: maintaining adatabase describing the results of executing a group of parametric testscenarios on a data-oriented large-scale software system after anupgrade from a first to a second installation, and before adjusting theparametric test scenarios to the upgrade; identifying an organizationfacing an upgrade from the first to the second installation and havingat least one parametric test scenario equivalent to at least one of theparametric test scenarios in the group; and based on the database,estimating results of executing the at least one equivalent parametrictest scenario by the organization on the upgraded system, withoutactually executing the at least one equivalent parametric test scenario.Optionally, based on the database, further suggesting the organizationchanges to adjust the at least one equivalent parametric test scenarioto the upgrade.

In one embodiment, a method for customization sharing based on similarcustomizations and usage including the following steps: identifyingfirst and second organizations having similar customizations and usagelogging 110; obtaining a customization unique to the first organizationthat is related to an in-use scenario; and suggesting adding theobtained customization to the second organization.

In one embodiment, a method for code sharing based on similarcustomizations and usage including the following steps: identifyingfirst and second organizations having similar customizations and usagelogging 110; obtaining user-generated code unique to the firstorganization that is related to an in-use scenario; and suggestingadding the obtained user-generated code to the second organization.

In this description, numerous specific details are set forth. However,the embodiments of the invention may be practiced without some of thesespecific details. In other instances, well-known hardware, software,materials, structures, and techniques have not been shown in detail inorder not to obscure the understanding of this description. In thisdescription, references to “one embodiment” mean that the feature beingreferred to may be included in at least one embodiment of the invention.Moreover, separate references to “one embodiment” or “some embodiments”in this description do not necessarily refer to the same embodiment.Illustrated embodiments are not mutually exclusive, unless so stated andexcept as will be readily apparent to those of ordinary skill in theart. Thus, the invention may include any variety of combinations and/orintegrations of the features of the embodiments described herein.Although some embodiments may depict serial operations, the embodimentsmay perform certain operations in parallel and/or in different ordersfrom those depicted. Moreover, the use of repeated reference numeralsand/or letters in the text and/or drawings is for the purpose ofsimplicity and clarity and does not in itself dictate a relationshipbetween the various embodiments and/or configurations discussed. Theembodiments are not limited in their applications to the details of theorder or sequence of steps of operation of methods, or to details ofimplementation of devices, set in the description, drawings, orexamples. Moreover, individual blocks illustrated in the figures may befunctional in nature and do not necessarily correspond to discretehardware elements. While the methods disclosed herein have beendescribed and shown with reference to particular steps performed in aparticular order, it is understood that these steps may be combined,sub-divided, or reordered to form an equivalent method without departingfrom the teachings of the embodiments. Accordingly, unless specificallyindicated herein, the order and grouping of the steps is not alimitation of the embodiments. Furthermore, methods and mechanisms ofthe embodiments will sometimes be described in singular form forclarity. However, some embodiments may include multiple iterations of amethod or multiple instantiations of a mechanism unless noted otherwise.For example, when an interface is disclosed in an embodiment, the scopeof the embodiment is intended to also cover the use of multipleinterfaces. Certain features of the embodiments, which may have been,for clarity, described in the context of separate embodiments, may alsobe provided in various combinations in a single embodiment. Conversely,various features of the embodiments, which may have been, for brevity,described in the context of a single embodiment, may also be providedseparately or in any suitable sub-combination. Embodiments described inconjunction with specific examples are presented by way of example, andnot limitation. Moreover, it is evident that many alternatives,modifications, and variations will be apparent to those skilled in theart. It is to be understood that other embodiments may be utilized andstructural changes may be made without departing from the scope of theembodiments. Accordingly, it is intended to embrace all suchalternatives, modifications, and variations that fall within the spiritand scope of the appended claims and their equivalents.

What is claimed is:
 1. A computer implemented method for identifyingmissing parametric test scenarios based on authorization policies,comprising: analyzing the authorization policies applied tonon-super-users of a specific data-oriented large-scale software systemin order to identify organization-specific in-use scenarios; whereby thenon-super-users are usually unauthorized to irrelevant operations;comparing the identified organization-specific in-use scenarios withavailable parametric test scenarios repairable by a non-programmer user,each available parametric test scenario repairable by a non-programmeruser tests at least two transactions, and at least 50% of instructionsdescribed in each parametric test scenario repairable by anon-programmer are expressed by user interaction activities; identifyingorganization-specific in-use scenarios that do not have correspondingparametric test scenarios repairable by a non-programmer user, referredto as first set of missing parametric test scenarios; and providing atleast one of the missing parametric test scenarios to a non-programmeruser, having proper authorizations, for generating a parametric testscenario repairable by a non-programmer user that covers the at leastone missing parametric test scenario and comprises instructionsexpressed by user interaction activities.
 2. The computer implementedmethod of claim 1, wherein the data-oriented large-scale software systemis Oracle ERP and the authorization policies are taken from OracleApplications User Management and Oracle Applications ResponsibilityManagement.
 3. The computer implemented method of claim 2, furthercomprising identifying the forms exposed to the non-super-users byanalyzing the responsibilities from Oracle Applications ResponsibilityManagement.
 4. The computer implemented method of claim 2, wherein thetransactions are selected from the group of forms, workflow, reports,concurrent programs, and other executable units.
 5. The computerimplemented method of claim 2, wherein the generated parametric testscenario repairable by a non-programmer defines different alternativesfor executing the test scenario.
 6. The computer implemented method ofclaim 5, wherein the parametric test scenario repairable by anon-programmer user further defines different alternatives forevaluating execution outcome, and is expressed in terms thatsubstantially conform to terms used in the non-programmer user's workenvironment, and coherent with actual manual operation of the testscenario.
 7. The computer implemented method of claim 1, furthercomprising identifying additional organization-specific in-use scenariosby registering the executed transactions and registering order ofexecuting the transactions.
 8. The computer implemented method of claim7, further comprising identifying the additional organization-specificin-use scenarios by registering context of executing the transactions.9. The computer implemented method of claim 1, wherein generating theparametric test scenario repairable by a non-programmer user comprisesrecording the non-programmer user executing the at least one missingtest scenario.
 10. A non-transitory computer readable medium for use ina computer to identify missing parametric test scenarios based onauthorization policies, the computer including a processor, and thenon-transitory computer readable medium comprising: program code foranalyzing the authorization policies applied to non-super-users of aspecific data-oriented large-scale software system in order to identifyorganization-specific in-use scenarios; whereby the non-super-users areusually unauthorized to irrelevant operations; program code forcomparing the identified organization-specific in-use scenarios withavailable parametric test scenarios repairable by a non-programmer user,each available parametric test scenario repairable by a non-programmeruser tests at least two transactions, and at least 50% of instructionsdescribed in each parametric test scenario repairable by anon-programmer are expressed by user interaction activities; programcode for identifying organization-specific in-use scenarios that do nothave corresponding parametric test scenarios repairable by anon-programmer user, referred to as first set of missing parametric testscenarios; and program code for providing at least one of the missingparametric test scenarios to a non-programmer user, having properauthorizations, for generating a parametric test scenario repairable bya non-programmer user that covers the at least one missing test scenarioand comprises instructions expressed by user interaction activities. 11.The non-transitory computer readable medium of claim 10, wherein thedata-oriented large-scale software system is Oracle ERP and theauthorization policies are taken from Oracle Applications UserManagement and Oracle Applications Responsibility Management.
 12. Thenon-transitory computer readable medium of claim 11, further comprisingprogram code for identifying the forms exposed to the non-super-users byanalyzing the responsibilities from Oracle Applications ResponsibilityManagement.
 13. The non-transitory computer readable medium of claim 11,wherein the transactions are selected from the group of forms, workflow,reports, concurrent programs, and other executable units.
 14. Thenon-transitory computer readable medium of claim 11, wherein theparametric test scenario repairable by a non-programmer user definesdifferent alternatives for executing the test scenario.
 15. Thenon-transitory computer readable medium of claim 14, wherein theparametric test scenario repairable by a non-programmer user furtherdefines different alternatives for evaluating execution outcome, and isexpressed in terms that substantially conform to terms used in thenon-programmer user's work environment, and coherent with actual manualoperation of the test scenario.
 16. The non-transitory computer readablemedium of claim 10, further comprising program code for identifyingadditional organization-specific in-use scenarios by registering theexecuted transactions and registering order of executing thetransactions.
 17. The non-transitory computer readable medium of claim16, further comprising program code for identifying the additionalorganization-specific in-use scenarios by registering context ofexecuting the transactions.
 18. The non-transitory computer readablemedium of claim 10, wherein generating the parametric test scenariorepairable by a non-programmer user comprises recording thenon-programmer user executing the at least one missing test scenario.19. A computer system to identify missing test scenarios based onauthorization policies, the computer system comprising: a storage devicecontaining computer usable code; and a processing unit operable toexecute the computer usable code to: analyze the authorization policiesapplied to non-super-users of a specific data-oriented large-scalesoftware system in order to identify organization-specific in-usescenarios; whereby the non-super-users are usually unauthorized toirrelevant operations; compare the identified organization-specificin-use scenarios with available parametric test scenarios repairable bya non-programmer user, each available parametric test scenariorepairable by a non-programmer user tests at least two transactions, andat least 50% of instructions described in each parametric test scenariorepairable by a non-programmer are expressed by user interactionactivities; identify organization-specific in-use scenarios that do nothave corresponding parametric test scenarios repairable by anon-programmer user, referred to as first set of missing parametric testscenarios; and provide at least one of the missing parametric testscenarios to a non-programmer user, having proper authorizations, togenerate a parametric test scenario repairable by a non-programmer userthat covers the at least one missing parametric test scenario andcomprises instructions expressed by user interaction activities.
 20. Thecomputer system of claim 19, wherein the data-oriented large-scalesoftware system is Oracle ERP and the authorization policies are takenfrom Oracle Applications User Management and Oracle ApplicationsResponsibility Management.
 21. The computer system of claim 19, whereinthe processing unit further executes the computer usable code toidentify the forms exposed to the non-super-users by analyzing theresponsibilities from Oracle Applications Responsibility Management. 22.The computer system of claim 19, wherein the transactions are selectedfrom the group of forms, workflow, reports, concurrent programs, andother executable units.
 23. The computer system of claim 20, wherein theparametric test scenario repairable by a non-programmer user definesdifferent alternatives for executing the test scenario.
 24. The computersystem of claim 23, wherein the parametric test scenario repairable by anon-programmer user further defines different alternatives forevaluating execution outcome, and is expressed in terms thatsubstantially conform to terms used in the non-programmer user's workenvironment, and coherent with actual manual operation of the testscenario.
 25. The computer system of claim 19, wherein the processingunit further executes the computer usable code to register the executedtransactions and register order of executing the transactions in orderto identify additional organization-specific in-use scenarios.
 26. Thecomputer system of claim 25, wherein the processing unit furtherexecutes the computer usable code to register context of executing thetransactions.